Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 901–1,000 of 1,004
Sutthipanyo, T., Lamsan, T., Thawornsusin, W., & Susutti, W. (2023) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon58879.2023.10322478 · Cited by 4
“Xu et al. [12] proposed using principal component analysis (PCA) for anomaly detection in distributed computing systems.”
MethodLogContrast: A Weakly Supervised Anomaly Detection Method Leveraging Contrastive Learning
Zhang, T., Huang, X., Zhao, W. P., Mo, G., & Bian, S. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00015 · Cited by 4
“The HDFS dataset [37] comprises 11,175,629 log entries generated by MapReduce jobs running on 203 Amazon EC2 nodes.”
MethodGrafting log analysis method of power data based on multi-layer clustering
Chang, Y., Yin, J., Liu, Z., Luo, F., Lin, J., Shen, Y., & He, S. (2023) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic59930.2023.10456321 · Cited by 4
“According to the generated log template, principal component analysis method is used, which can reduce the analysis complexity and detect outliers [2].”
MethodMulti-Stage Defense: Enhancing Robustness in Sequence-Based Log Anomaly Detection
Tan, K., Zhan, D., Yu, Z., Ye, L., Zhang, H., & Fang, B. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622485 · Cited by 4
“We evaluate our approach on two public datasets, HDFS [12] and Blue Gene/L [13].”
MethodTinyLog: Log Anomaly Detection with Lightweight Temporal Convolutional Network for Edge Device
Meng, C., & Chen, N. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651312 · Cited by 3
“Currently, researchers have proposed many methods for log anomaly detection, which are mainly categorized into the following two groups: machine learning methods such as Principal Component Analysis(PCA)[1], Support Vector Machine[2] and K-nearest Neighbor[3].”
MethodEfficient Log Parsing Method Based on Log Locality Features
Wen, J., Ma, C., Xie, K., He, S., Pei, S., Chen, Y., Liang, W., & Feng, Q. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2024.9010192 · Cited by 3
Automation of Log Collection and Incident Triage for Security Operations: A Review
Okwesa, I. J., Ajiga, D., & Iwuanyanwu, U. A. (2019) · Iconic Research and Engineering Journals · doi:10.64388/irev3i5-1722629 · Cited by 4
Narrowing Down Possible Causes of Performance Anomaly in Web Applications
Iwata, S., & Kono, K. (2010) · European Dependable Computing Conference · doi:10.1109/edcc.2010.30 · Cited by 3
“[8] have developed log-mining techniques to automate log analysis.”
MethodThe Method of Analyzing Firewall Log Data using MapReduce based on NoSQL
Choi, B., Kong, J.-H., Hong, S.-S., & Han, M.-M. (2013) · 정보보호학회논문지 · doi:10.13089/jkiisc.2013.23.4.667 · Cited by 3
Compatibility testing using patterns-based trace comparison
Ranganath, V.-P., Vallathol, P., & Gupta, P. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642942 · Cited by 3
“Beyond software testing, there have been numerous efforts [13,14,18–21 , 23 In a similar vein, Barringer et al. used human specified quantified (parameterized) temporal patterns with logs to enable postmortem runtime verification of flight software for NASA’s recent Mars rover mission [7].”
MethodMi, H., Wang, H., Yin, G., Cai, H., Zhou, Q., & Sun, T. (2011) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsnw.2011.5958799 · Cited by 3
VisRAID: Visualizing Remote Access for Intrusion Detection
Trethowen, L., Anslow, C., Marshall, S., & Welch, I. S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19962-7_17 · Cited by 3
“present a system log data mining application [13].”
BackgroundPerformance Analysis Tool for HPC and Big Data Applications on Scientific Clusters
Yoo, W., Koo, M. S., Cao, Y., Sim, A., Nugent, P., & Wu, K. J. (2016) · Springer eBooks · doi:10.1007/978-3-319-33742-5_7 · Cited by 4
On high efficient cloud video transcoding
Huang, J.-C., Wu, C.-Y., & Chen, J. (2015) · International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) · doi:10.1109/ispacs.2015.7432759 · Cited by 3
“After these working nodes finishing corresponding small jobs, the Reducer perform the integration process [4] to render the big job (output stage).”
Logging Analysis and Prediction in Open Source Java Project
Lal, S., Sardana, N., & Sureka, A. (2018) · Advances in systems analysis, software engineering, and high performance computing book series · doi:10.4018/978-1-5225-5314-4.ch003 · Cited by 3
Creating the knowledge about IT events
Barash, G., Cohen, I. L., Mordechai, E., Staelin, C., & Dakar, R. (2010) · Cited by 3
Understanding and Debugging Complex Software Systems: A Data-Driven Perspective.
Yu, X. (2018) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 3
On the Use of Automated Log Clustering to Support Effort Reduction in Continuous Engineering
Rosenberg, C. M., & Moonen, L. (2018) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2018.00032 · Cited by 3
“[26] demonstrate a technique for automatically identifying anomalous events in large system logs.”
MethodAn efficient real-time data collection framework on petascale systems
Huang, H., Zhou, L., Lu, Y., Xiao, T. Y., Leng, C., Li, C., & Quan, Z. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.039 · Cited by 3
Wiedemann, N., Kammler, F., Varwig, A., & Thomas, O. (2019) · Journal of the Association for Information Systems · Cited by 3
An Efficient Massive Log Discriminative Algorithm for Anomaly Detection in Cloud
Liu, J., Li, J., & Wu, C. (2019) · Globecom · doi:10.1109/globecom38437.2019.9013839 · Cited by 3
“We first extract the raw logs of normal execution which block id is label normal in the dataset [9, 21], then execute the log sorting module, finally parse log entries into a log key sequence to get the experimental dataset.”
BackgroundMethodLog Anomaly Detection Based on Bi-LSTM Feature Extraction
Zhang, Y., Zhang, D., Guo, F., Wang, X., Duan, Y., & Zhang, X. (2022) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit55514.2022.9943901 · Cited by 3
“Typical algorithms include PCA [1], Log Clustering [2], Decision tree [3], etc.”
MethodLog Analysis for Failure Diagnosis and Workload Prediction in Cloud Computing
Hunt, K. (2016) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 3
Towards a Taxonomy of Software Log SmellsPreprint
Saarimäki, N., Yazdi, Z. G., Shin, D., & Bianculli, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.09284 · Cited by 3
“→ Reduces the reliability of automated processing and parsing.”
A posteriori log analysis and security rules violation detection
Dernaïka, F. (2020) · theses.fr (ABES) · doi:10.70675/e0e4cc1ez9d1dz494cz9b0dz7b31791ba53f · Cited by 3
Network Event Extraction from Log Data with Nonnegative Tensor Factorization
Kimura, T., Ishibashi, K., Mori, T., Sawada, H., Toyono, T., Nishimatsu, K., Watanabe, A., Shimoda, A., & Shiomoto, K. (2017) · IEICE Transactions on Communications · doi:10.1587/transcom.2016ebp3430 · Cited by 2
TransFlowLog: Log Anomaly Detection Based on Transformer Encoder and Interflow Decoder
Lin, Z., Lu, S., Han, N., Wang, D., Wei, X., & Wang, M. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00196 · Cited by 3
AML: An accuracy metric model for effective evaluation of log parsing techniques
Sedki, I., Hamou‐Lhadj, A., & Mohamed, O. A. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112154 · Cited by 3
“Logs, however, are partially structured [5, 11], making the extraction of useful information from massive log files a challenging task [16, 17].”
BackgroundTWLog: Task Workflow-Based Log Anomaly Detection
Zhang, S., Fan, D., He, L., Liu, Y., & Chen, D. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7244-5_1 · Cited by 5
EDSLog: Efficient Log Anomaly Detection Method Based on Dataset Partitioning
Liang, F., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0602-3_22 · Cited by 2
Wang, P., Zhang, X., & Cao, Z. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-024-06850-y · Cited by 2
“Feature extraction vectorizes the log template into template index or semantic [4–6].”
LogSage: Log Summarization Assistant with Guided Enhancement
Lu, Y. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707990 · Cited by 3
Characterizing Logs in Vulnerability Reports: In-Depth Analysis and Security Implications
Shu, Y., Zheng, L., Chen, J., & Xuan, J. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00072 · Cited by 2
“In software security, logs are widely applied in anomaly detection tasks [27, 58, 59, 38, 15, 35, 12, 21].”
Wu, X., Li, H., & Khomh, F. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00527-3 · Cited by 5
“Both supervised [20–22] and unsupervised [8] classifiers can be used under this formulation.”
InfluentialFelix, A. O. (2025) · International Journal of Future Engineering Innovations · doi:10.54660/ijfei.2025.2.4.71-84 · Cited by 4
Arabov, M. K., Burnashev, R. A., & Medvedeva, O. A. (2025) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon65989.2025.11177379 · Cited by 3
“Later, Xu et al. developed a method for clustering console logs followed by frequent pattern mining to detect anomalous behaviour [4].”
MHP-RCA: Multivariate Hawkes Process-based Root Cause Analysis in microservice systems
Zhang, Z., Wang, J., Li, B., Liu, Y., Wu, H., & Hung, P. C. K. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107938 · Cited by 2
ZeroLog: Zero-Label Generalizable Cross-System Log-based Anomaly Detection
Zhao, X., Jia, T., He, M., Li, Y., & Huang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00018 · Cited by 3
“We conducted systematic experiments on three publicly available log datasets: HDFS [28], BGL [29] and OpenStack [1].”
GLU-PM: A Pretrained Model Toward Generalized Log Understanding
Liu, X., Zheng, S., Zou, H., Yu, H., & Gao, S. (2025) · IEEE Transactions on Reliability · doi:10.1109/tr.2025.3642425 · Cited by 2
“In response, automated log analysis methods based on machine learning [6], [7]anddeeplearning[8],[9],[10],[11]haveachievedsignificant progress.”
DualBERT: Fusing Symbolic and Temporal Dynamics for High-Precision Log Anomaly Detection
Alzamil, A. A., Guesmi, R., AlGhadhban, A., Guesmi, T., & Kachouri, A. (2026) · IEEE Access · doi:10.1109/access.2026.3653996 · Cited by 2
HPCLOG: A Transformer-Based Log Anomaly Detection Framework for Distributed Systems
Shi, Q., Yang, Z., & Shao, S. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323005 · Cited by 2
“We conduct experiments with our framework using two public system log datasets: the HDFS dataset [17] and the OpenStack dataset [8], which are popular distributed system log datasets widely used for LAD.”
Krone: Hierarchical and Modular Log Anomaly Detection
Ma, L., Liu, J., Zhang, T., VanNostrand, P. M., Hofmann, D. M., Cao, L., Rundensteiner, E. A., & Chen, J. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00135 · Cited by 3
“As in Figure 3 Nested-Aware Detector via LLM Reasoning.”
LogBoost: Boost Log Anomaly Detection by Cherry-Picking Log Sequences
Li, M., Li, X., Chen, P., Lai, Y., & Zheng, Z. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3664358 · Cited by 3
Unseen Anomaly Detection from System Logs
Tang, Y., Zhang, Z., Fang, L., Link, S., Chen, W., & Zhao, K. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3786705 · Cited by 2
Gupta, V., Jain, N., Anand, A., Jain, D., Yadav, M. P., & Singh, Y. P. (2026) · IEEE Access · doi:10.1109/access.2026.3682603 · Cited by 2
“Initial techniques were based on manually curated templates, threshold-based alerts, and pattern-matching engines, implicitly relying on that the failure signatures were both stable, enumerable, and time-localized [6], [7].”
LogSieve: Task-Aware CI Log Reduction for Sustainable LLM-Based Analysis
Barnes, M. E., Ghaleb, T. A., & Hassan, S. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793380 · Cited by 5
Shatnawi, M. Q., & Hefeeda, M. M. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806275 · Cited by 2
“Also, the creation of a failure predictor is complex and time consuming and thus not suitable for real-time management [25, 12].”
BackgroundClustering Performance Anomalies Based on Similarity in Processing Time Changes
Iwata, S., & Kono, K. (2012) · IPSJ Online Transactions · doi:10.2197/ipsjtrans.5.1 · Cited by 2
“We use an existing anomaly detector [4], [5], [6], [7], [8], [9], [10], [11] as a back-end in our clustering method to collect the measurements and detect anomalies.”
MethodLogging Inter-Thread Data Dependencies in Linux Kernel
KUBOTA, T., Aota, N., & Kono, K. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edp7255 · Cited by 2
“Logs are helpful for understanding the runtime behavior of software systems, and are widely used in software diagnosis and management, for example, in error debugging [1], [2], performance analysis [3], anomaly detection [4], [5], work-load modeling [6], and system monitoring [7], [8].”
BackgroundMethodPerfDB: A Data Management System for Fine-Grained Performance Anomaly Detection
Kimball, J., Lima, R. A., Kanemasa, Y., & Pu, C. (2020) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic50333.2020.00021 · Cited by 2
A new Multi-Agents System based on Blockchain for Prediction Anomaly from System Logs
Binlashram, A., Bouricha, H., Hsairi, L., & Ahmadi, H. A. (2020) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/3428757.3429149 · Cited by 4
“More information about this dataset can be found in [29].”
BackgroundResultInfluentialKumar, G. R., Karthik, J., Rao, B. S., & Prasad, C. (2022) · International Conference on Electronics and Renewable Systems (ICEARS) · doi:10.1109/icears53579.2022.9751876 · Cited by 2
RetractedImproving Logging Prediction on Imbalanced Datasets: A Case Study on Open Source Java Projects
Lal, S., Sardana, N., & Sureka, A. (2020) · IGI Global eBooks · doi:10.4018/978-1-7998-2460-2.ch039 · Cited by 2
Research on network cloud equipment anomaly and root cause analysis
Zou, D., Ding, J., Wang, X., Ye, X., & Ouyang, Y. (2022) · ITU Journal on Future and Evolving Technologies · doi:10.52953/tvlo2995 · Cited by 3
Extreme Gradient Boosting based Anomaly detection for Kubernetes Orchestration
Yolchuyev, A. (2023) · doi:10.1109/it57431.2023.10078576 · Cited by 2
“Several research have examined [10, 11] the analysis and detection of application or system logs.”
BackgroundAn Efficient Way to Parse Logs Automatically for Multiline Events
Yu, M., & Zhang, X. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.037505 · Cited by 2
The Sub-Sequence Summary Method for Detecting Anomalies in Logs
Horväth, G., Kádár, A., & Szilágyi, P. (2023) · IEEE Access · doi:10.1109/access.2023.3266990 · Cited by 4
“In [16] two kinds of features are extracted from the log file: a count vector storing the number of occurrences of the different log line templates, and the ratio of the different values of state variables in a time window.”
MethodInfluentialPECULIARITIES OF DATA LOGGING AND INFLUENCE OF APPLICATION TYPE ON THE CHOICE OF LOGGING METHODOLOGY
KAPITANETS, S., & Радельчук, Г. (2022) · Herald of Khmelnytskyi National University Technical sciences · doi:10.31891/2307-5732-2022-315-6-98-101 · Cited by 2
Jiang, Z., Gao, Y., Yuan, J., Yuan, K., & Li, X. (2023) · International Conference on Consumer Electronics and Computer Engineering (ICCECE) · doi:10.1109/iccece58074.2023.10135232 · Cited by 2
A Log Parsing Framework for ALICE O2 Facilities
Marlaithong, T., Barroso, V. C., & Phunchongharn, P. (2023) · IEEE Access · doi:10.1109/access.2023.3293406 · Cited by 2
“Various data mining techniques are applied to the template to build an automated log analysis, namely anomaly detection [3], [4], network security [5], fault diagnosis [6], etc. Numerous studies have been conducted on log parsing approaches that can be implemented across a wide range of system…”
MethodTowards a Classification of Log Parsing Errors
Sedki, I., Hamou‐Lhadj, A., Mohamed, O. A., & Ezzati‐Jivan, N. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00023 · Cited by 2
“The analysis of logs can help with many software engineering tasks including debugging [1] [2], detection of data leakage [3], and anomaly detection [4] [5].”
MethodPolo: Adaptive Trie-Based Log Parser for Anomaly Detection
Zhou, Y., & Su, Y. (2023) · Mathematics · doi:10.3390/math11234797 · Cited by 3
Cognition: Accurate and Consistent Linear Log Parsing Using Template Correction
Tian, R., Diao, Z., Jiang, H., & Xie, G. (2023) · Journal of Computer Science and Technology · doi:10.1007/s11390-021-1691-3 · Cited by 2
Log Analysis For Network Attack Detection Using Deep Learning Models
Pham, H. P., Nguyen, V. H., Tran, N. A., & Nguyen, M. H. (2023) · International Symposium on Information and Communication Technology · doi:10.1145/3628797.3628943 · Cited by 2
“One widely used log dataset is HDFS [19], [20], which contains over 11 million log messages from a Hadoop Distributed File System deployed on Amazon EC2.”
MethodMachine Learning and Anomaly Detection for an Automated Monitoring of Log Data
Falzone, S., Gühring, G., & Jung, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-57452-8_13 · Cited by 3
Gloss: Guiding Large Language Models to Answer Questions from System Logs
Huang, S., Liu, Y., Qi, J., Shang, J., Xiao, Z., Fung, C., Wu, Z., Yang, H., Luan, Z., & Qian, D. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00017 · Cited by 5
“Some of the datasets in LogHub were released by previous studies such as HDFS [16], Hadoop [17], and BGL [18].”
MethodAnálise de desempenho de Log Parsers da coleção Logpai em dados brutos de dispositivos Android
Bessa, J. A., Filho, R., Souza, G., Pessoa, L., Barreto, R., & de Freitas, R. (2024) · doi:10.5753/wperformance.2024.2423 · Cited by 3
A Review of Software Testing Process Log Parsing and Mining
Xue, K., Han, Q., Sheng, H., Shi, Z., & Qiao, Y. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00055 · Cited by 4
“Xu et al. [15] observed that within Google’s systems, thousands of new log statements are introduced monthly, underscoring the critical nature of log parsing.”
BackgroundMethodSeLog: A Log Anomaly Detection Method on Log Event and Variable Semantic
Li, C., Chen, G., Mi, X., Zhang, Y. E., & Zhou, H. (2024) · IEEE International Conference on Information Systems and Computer Aided Education (ICISCAE) · doi:10.1109/iciscae62304.2024.10761356 · Cited by 2
“Reference [4] proposes using PCA to achieve log anomaly detection.”
Semi-supervised Configuration and Optimization of Anomaly Detection Algorithms on Log Data
Beck, V., Landauer, M., Wurzenberger, M., Skopik, F., & Rauber, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825202 · Cited by 2
“Most publications assess their approaches using well-known datasets such as HDFS, BGL, Thunderbird, OpenStack or Hadoop which are publicly available on LogHub [18], [19].”
LogLSHD: Fast Log Parsing with Locality-Sensitive Hashing and Dynamic Time Warping
Huang, S.-W., Wu, X., & Li, H. (2025) · doi:10.1145/3727582.3728685 · Cited by 3
Unsupervised Automata Learning via Discrete Optimization
Lutz, S., Kaminskyi, D., Wittbold, F., Dierl, S., Howar, F. M., König, B., Müller, E., & Neider, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04587-4_9 · Cited by 2
“We evaluated all three learning settings in the context of anomaly detection on three datasets: a modified version of the ALFRED benchmark set [54] and the two real-world log datasets HDFS [58] and BGL [44], provided by the Loghub system log dataset collection [59].”
Zhao, X., Jia, T., He, M., & Li, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786811 · Cited by 2
Thota, M. R. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2221193 · Cited by 2
Vollem, S. (2023) · International Journal of Future Innovative Science and Technology · doi:10.15662/ijfist.2023.0603003 · Cited by 2
Automated performance characterization of applications using hardware monitoring events
Yoo, W. (2012) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 2
A next generation entropy based framework for alert detection in system logs
Makanju, A., Zincir-Heywood, N., & Milios, E. (2011) · IFIP/IEEE International Symposium on Integrated Network Management (IM) · doi:10.1109/inm.2011.5990587 · Cited by 2
Chen, J., Xu, X., Osterweil, L. J., Zhu, L., Brun, Y., Bass, L., Xiao, J., Li, M., & Wang, Q. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.08.043 · Cited by 2
Grzegorczyk, K., Baggiolini, V., & Krzysztof, Z. (2014) · Computer Science · doi:10.7494/csci.2014.15.4.351 · Cited by 2
“at Berkeley in the late 2000 s [17].”
IOSIG+: On the Role of I/O Tracing and Analysis for Hadoop Systems
Feng, B., Yang, X., Feng, K., Yin, Y., & Sun, X. (2015) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster.2015.17 · Cited by 2
“Because loggers in Hadoop are added by random independent developers, they are incomplete, less reliable, and even misleading especially for I/O behavior analysis [11], [12].”
BackgroundClassification of Post-deployment Performance Diagnostic Techniques for Large-scale Software Systems
Malik, H., & Shakshuki, E. (2014) · Procedia Computer Science · doi:10.1016/j.procs.2014.08.036 · Cited by 3
Analytics for resiliency in the mainframe
Ein‐Dor, L., Goldschmidt, Y., Lavi, O., Miller, G. E., Ninio, M., & Dillenberger, D. E. (2013) · IBM Journal of Research and Development · doi:10.1147/jrd.2013.2263891 · Cited by 2
A Dependable Monitoring Mechanism Combining Static and Dynamic Anomaly Detection for Network Systems
Wang, G., Chen, S., Zhou, Z., & Lin, M. (2014) · International Journal of Future Generation Communication and Networking · doi:10.14257/ijfgcn.2014.7.1.01 · Cited by 2
“…kernel level, such as analyzing system call sequence [14], analyzing running status of OS kernel [15], etc. Non-kernel monitoring judges whether the system is anomalous by analyzing OS operation logs [16], read and write operations of the file system, as well as status of network connection, etc.”
BackgroundCharacterizing Incidents in Cloud-Based IoT Data Analytics
Truong, H., & Halper, M. (2018) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2018.00068 · Cited by 2
“Similarly, we can utilize existing techniques for processing logs [15] to extract information for incident analytics, but these works are not for incidents in big data analytics ML techniques for detecting errors has been widely used.”
MethodVisualization of Server Log Data for Detecting Abnormal Behaviour
Suman, R., Far, B. H., Mohammed, E. A., Nair, A., & Janbakhsh, S. (2018) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri.2018.00044 · Cited by 2
“Jordon [5] were detecting system problems by mining console logs.”
BackgroundAnomaly Detection in POSTFIX mail log using Principal Component Analysis
Tran, C.-P., & Tran, D.-K. (2018) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse.2018.8573410 · Cited by 2
“Below is the formula to calculate the distance from a data point y to the normal subspace made by k dimensions [5] [7] [8]:”
BackgroundMethodInfluentialReanzi, S. R. D., Rajiah, V., & Thangaiah, P. R. J. (2018) · IEEE International Conference on Cloud Computing in Emerging Markets (CCEM) · doi:10.1109/ccem.2018.00026 · Cited by 2
“The console logs mining helps detect problems in large scale systems [5].”
BackgroundFARE: Schema-Agnostic Anomaly Detection in Social Event Logs
Shah, N. (2019) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2019.00049 · Cited by 2
“[13] featurizes console logs based on counts of event classes in a trace, and uses offline principal component analysis to identify resources with odd class distributions.”
MethodForecasting health of complex IT systems using system log data
Patel, S. S. (2020) · Journal of Banking and Financial Technology · doi:10.1007/s42786-019-00011-z · Cited by 2
“In related literature, there are the models, based on pattern recognition [1] and others which used principal component analysis (PCA) to detect anomalies from parsed textual log messages [2–4].”
MethodA Logging Overhead Optimization Method Based on Anomaly Detection Model
Wang, Y., & Zheng, Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-70626-5_37 · Cited by 2
Failure Avoidance through Fault Prediction Based on Synthetic Transactions
Shatnawi, M. Q., & Ripeanu, M. (2011) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid.2011.61 · Cited by 2
“In their research, Xu et al. [7, 9] also explain the voluminous nature of data logs, and claim that logs are not actually helpful in a lot of cases, simply because of their large scale volume.”
BackgroundMethodInfluentialSurvey on Log Clustering Approaches
Skopik, F., Wurzenberger, M., & Landauer, M. (2021) · Springer eBooks · doi:10.1007/978-3-030-74450-2_2 · Cited by 3
Neto, W. L., & Filho, I. D. M. B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87013-3_3 · Cited by 2
A mixture modeling approach for clustering log files with coreset and user feedback
Bunker, J., Curtis, K., Girolami, M., & Sriharsha, R. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.01.027 · Cited by 2
Yue, G., sun, Z., Tian, J., Zhu, H., & Zhang, B. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-0390-8_6 · Cited by 2
Detecting anomalous user behavior from NGINX web server logs
Benova, L., & Hudec, L. (2022) · Zooming Innovation in Consumer Technologies Conference (ZINC) · doi:10.1109/zinc55034.2022.9840589 · Cited by 2
Maia, G. V., Coutinho, T., Gonçalves, E. B., Silva, G. R. L., Mendes, E. M. A. M., Mendes, M. M. A. M., Caetano, S. R., Mitt, G. M., & Braga, A. P. (2022) · Journal of Network and Systems Management · doi:10.1007/s10922-022-09671-7 · Cited by 2
“…different strategies by focusing on device logs to identify anomalies [7], by presenting new methodologies to extract representative information from network logs aiming at machine learning models [8] and by adopting causal inference to identify network abnormal events and their root causes [9].”
MethodA Heuristic Approach Using Template Miners for Error Prediction in Telecommunication Networks
Marjai, P., Lehotay–Kéry, P., & Kiss, A. (2022) · IEEE Access · doi:10.1109/access.2022.3221427 · Cited by 2
“The second phase is the feature extraction, during which correlating messages are grouped together, and represented with a vector per group, to create the message count vector [32].”
BackgroundA New Distributed Log Anomaly Detection Method based on Message Middleware and ATT-GRU
Fang, W., Jia, X., Zhang, W., & Sheng, V. S. (2023) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2023.02.011 · Cited by 2
Development of Anomaly Detection System Based on Distributed Log Tracing
Khudyakov, D. A. (2023) · Vestnik NSU Series Information Technologies · doi:10.25205/1818-7900-2023-21-1-62-72 · Cited by 2
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.