The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Experience Report: System Log Analysis for Anomaly Detection
Abstract
Anomaly detection plays an important role in management of modern large-scale distributed systems. Logs, which record system runtime information, are widely used for anomaly detection. Traditionally, developers (or operators) often inspect the logs manually with keyword search and rule matching. The increasing scale and complexity of modern systems, however, make the volume of logs explode, which renders the infeasibility of manual inspection. To reduce manual effort, many anomaly detection methods based on automated log analysis are proposed. However, developers may still have no idea which anomaly detection methods they should adopt, because there is a lack of a review and comparison among these anomaly detection methods. Moreover, even if developers decide to employ an anomaly detection method, re-implementation requires a nontrivial effort. To address these problems, we provide a detailed review and evaluation of six state-of-the-art log-based anomaly detection methods, including three supervised methods and three unsupervised methods, and also release an open-source toolkit allowing ease of reuse. These methods have been evaluated on two publicly-available production log datasets, with a total of 15,923,592 log messages and 365,298 anomaly instances. We believe that our work, with the evaluation results as well as the corresponding findings, can provide guidelines for adoption of these methods and provide references for future development.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Data mining: concepts and techniques
(2012) · Choice Reviews Online · doi:10.5860/choice.49-3305 · Cited by 28,531
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 10×
“Fortunately, by exploring an abundance of literature and intensively contacting the corresponding authors, we have successfully obtained two log datasets, HDFS data [47] and BGL data [36], which are suitable for evaluating existing anomaly detection methods.”
BackgroundMethodInfluentialAkidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824076 · Cited by 750
What Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J.-G., Wang, Y., & Li, J. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.60 · Cited by 542
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453
Failure diagnosis using decision trees
Chen, M., Zheng, A. X., Lloyd, J., Jordan, M. I., & Brewer, E. (2004) · International Conference on Autonomic Computing, 2004. Proceedings. · doi:10.1109/icac.2004.1301345 · Cited by 433
Log clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J.-G., Zhang, Y., & Chen, X. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2889232 · Cited by 427
Clustering event logs using iterative partitioning
Makanju, A. A. O., Zincir-Heywood, A. N., & Milios, E. E. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557154 · Cited by 338
Mining invariants from console logs for system problem detection
Lou, J., Fu, Q., Yang, S., Xu, Y., & Li, J. (2010) · Cited by 302
Failure Prediction in IBM BlueGene/L Event Logs
Liang, Y., Zhang, Y., Xiong, H., & Sahoo, R. (2007) · Seventh IEEE International Conference on Data Mining (ICDM 2007) · doi:10.1109/icdm.2007.46 · Cited by 292
Advances and challenges in log analysis
Oliner, A., Ganapathi, A., & Xu, W. (2012) · Communications of the ACM · doi:10.1145/2076450.2076466 · Cited by 290
Fingerprinting the datacenter: automated classification of performance crises
Bodik, P., Goldszmidt, M., Fox, A., Woodard, D. B., & Andersen, H. (2010) · Proceedings of the 5th European conference on Computer systems · doi:10.1145/1755913.1755926 · Cited by 250
LogSig: generating system events from raw textual logs
Tang, L., Li, T., & Perng, C.-S. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063690 · Cited by 244
Where do developers log? an empirical study on logging practices in industry
Fu, Q., Zhu, J., Hu, W., Lou, J.-G., Ding, R., Lin, Q., Zhang, D., & Xie, T. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591175 · Cited by 230
Self-citationAn Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2016) · 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2016.66 · Cited by 230
Self-citationLeveraging existing instrumentation to automatically infer invariant-constrained models
Beschastnikh, I., Brun, Y., Schneider, S., Sloan, M., & Ernst, M. D. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025151 · Cited by 216
Learning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R., & Zhang, D. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.60 · Cited by 212
Self-citationEvaluating Computer Intrusion Detection Systems: A Survey of Common Practices
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A., & Payne, B. D. (2015) · ACM Computing Surveys · doi:10.1145/2808691 · Cited by 194
Mi, H., Wang, H., Zhou, Y., Lyu, M. R.-T., & Cai, H. (2013) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.21 · Cited by 176
Self-citationBe conservative: enhancing failure diagnosis with proactive logging
Yuan, D., Park, S., Huang, P., Liu, Y., Lee, M. M., Tang, X., Zhou, Y., & Savage, S. (2013) · Cited by 143
Automated Identification of Failure Causes in System Logs
Mariani, L., & Pastore, F. (2008) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2008.48 · Cited by 130
Assisting developers of Big Data Analytics Applications when deploying on Hadoop clouds
Shang, W., Jiang, Z. M., Hemmati, H., Adams, B., Hassan, A. E., & Martin, P. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606586 · Cited by 130
Farshchi, M., Schneider, J.-G., Weber, I. M., & Grundy, J. (2015) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2015.7381796 · Cited by 116
SALSA: analyzing logs as state machines
Tan, J., Pan, X., Kavulya, S. P., Gandhi, R., & Narasimhan, P. (2008) · Cited by 95
Industry Practices and Event Logging: Assessment of a Critical Software Development Process
Pecchia, A., Cinque, M., Carrozza, G., & Cotroneo, D. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.145 · Cited by 93
LogMaster: Mining Event Correlations in Logs of Large-Scale Cluster Systems
Fu, X., Ren, R., Zhan, J., Zhou, W. J., Jia, Z. Y., & Lü, G. (2012) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2012.40 · Cited by 90
Experience report: Evaluating the effectiveness of decision trees for detecting code smells
Amorim, L., Costa, E. D. B., Antunes, N., Fonseca, B., & Ribeiro, M. (2015) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2015.7381819 · Cited by 85
Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays
Nagappan, M., Wu, K. J., & Vouk, M. A. (2009) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.23 · Cited by 69
Improving Log-based Field Failure Data Analysis of multi-node computing systems
Pecchia, A., Cotroneo, D., Kalbarczyk, Z., & Iyer, R. K. (2011) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2011.5958210 · Cited by 65
An empirical study of long lived bugs
Saha, R. K., Sarfraz Khurshid, D., & Perry, D. E. (2014) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/csmr-wcre.2014.6747164 · Cited by 52
Log-Based Reliability Analysis of Software as a Service (SaaS)
Banerjee, S. K., Srikanth, H., & Čukić, B. (2010) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2010.46 · Cited by 52
Babenko, A., Mariani, L., & Pastore, F. (2009) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1572272.1572300 · Cited by 51
An empirical study of injected versus actual interface errors
Lanzaro, A., Natella, R., Winter, S., Cotroneo, D., & Suri, N. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610418 · Cited by 44
Applying Data Mining for Detecting Anomalies in Satellites
Azevedo, D. R., Ambrósio, A. M., & Vieira, M. P. A. (2012) · European Dependable Computing Conference · doi:10.1109/edcc.2012.19 · Cited by 23
What Logs Should You Look at When an Application Fails? Insights from an Industrial Case Study
Cinque, M., Cotroneo, D., Corte, R. D., & Pecchia, A. (2014) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2014.69 · Cited by 7
Research works printed but not linked: 10
- 0, Apache SPARK
- 0, Apache Hadoop
- 10.1109/TDSC.2014.2334305 · doi:10.1109/tdsc.2014.2334305
- 10.1109/DSNW.2013.6615513 · doi:10.1109/dsnw.2013.6615513
- 10.1145/2351676.2351735 · doi:10.1145/2351676.2351735
- 0, Log Management Service in the Cloud-Loggly
- 0, Logentries Log management & analysis software made easy
- alonso, 2011, Avresky. Predicting software anomalies using machine learning techniques, NCA' 11 Proc of the 10th IEEE International Symposium on Network Computing and Applications, 163
- 10.1145/2600176.2600205 · doi:10.1145/2600176.2600205
- salton, 1987, Term weighting approaches in automatic text retrival, Technical Report
Cited by
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Machine Learning for Anomaly Detection: A Systematic Review
Nassif, A. B., Talib, M. A., Nasir, Q., & Dakalbab, F. M. (2021) · IEEE Access · doi:10.1109/access.2021.3083060 · Cited by 748
LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs
Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P., et al. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/658 · Cited by 708
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
Self-citationLogBERT: Log Anomaly Detection via BERT
Guo, H., Yuan, S., & Wu, X. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534113 · Cited by 454
Metadata
- Authors
- He, S., Zhu, J., He, P., & Lyu, M. R.
- Venue
- 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)
- Publisher
- IEEE
- Published
- 2016-10
- Pages
- 207-218
- OpenAlex
- W2560021099
- Cited by
- 593
- References
- 40
- FWCI
- 20.91
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.