Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.

Detecting large-scale system problems by mining console logs

Xu, Wei, Huang, Ling, Fox, Armando, Patterson, David, Jordan, Michael I. · 2009 · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles

Abstract

Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/1629575.1629587

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 16

  1. Borthakur D., 2007, Hadoop Project Website
  2. Chen M.Y., 2004, Proc. NSDI'04, 23
  3. Dunia R., 1997, Proc. ACC
  4. Fonseca R., 2007, In Proc. NSDI
  5. Hansen S.E., 1993, Proc. USENIX LISA '93, 145
  6. Hatcher E., 2004, Manning Publications Co.
  7. Jiang W., 2009, Proceedings of USENIX FAST'09
  8. Jolliffe I., 2002, Springer
  9. Lim C., 2008, Proc. DSN
  10. Ma S., 2001, Proc. IEEE ICDE
  11. 10.5555/1394399 · doi:10.5555/1394399
  12. Prewett J.E., 2003, Proc. Annual Conf. on Linux Clusters
  13. Salton G., 1987, Cornell
  14. 10.5555/1111682.1111732 · doi:10.5555/1111682.1111732
  15. Tan J., 2008, Proc. of WASL '08
  16. Witten I.H., 2000, Morgan Kaufmann

Cited by

  • DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning

    Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835

    “Principal Component Analysis (PCA) method [39] assumes that there are different “sessions” in a log file that can be easily identified by a session id attached to each log entry.”
    BackgroundMethodInfluential
  • Drain: an online log parsing approach with fixed depth tree

    He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943

  • Robust log-based anomaly detection on unstable log data

    Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761

    “Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
    BackgroundMethodResultInfluential
  • MapReduce online

    Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673

    “MapReduce is often used to analyze streams of constantly-arriving data, such as URL access logs [6] and system console logs [28].”
    Method
  • Experience Report: System Log Analysis for Anomaly Detection

    He, S., Zhu, J., He, P., & Lyu, M. R. (2016) · 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2016.21 · Cited by 593

    “Fortunately, by exploring an abundance of literature and intensively contacting the corresponding authors, we have successfully obtained two log datasets, HDFS data [47] and BGL data [36], which are suitable for evaluating existing anomaly detection methods.”
    BackgroundMethodInfluential

See all 1,332

Metadata

Authors
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
Affiliations
  • University of California, Berkeley (US)
  • Intel (United States) (US)
Venue
Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
Publisher
ACM
Published
2009-10-11
Pages
117-132
OpenAlex
W2039157918
Cited by
1,332
References
44
Citations by year
2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
FWCI
31.53
Type
conference paper
Language
en
Open access
closed
Locations
ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Detecting large-scale system problems by mining console logs. https://app.cochat.ai/papers/10.1145/1629575.1629587
The paper
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009). Detecting large-scale system problems by mining console logs. Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. https://doi.org/10.1145/1629575.1629587

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.