Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 501–600 of 1,004
PILAR: Studying and Mitigating the Influence of Configurations on Log Parsing
Dai, H., Tang, Y., Li, H., & Shang, W. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00077 · Cited by 21
Practical Anomaly Detection over Multivariate Monitoring Metrics for Online Services
Liu, J., Yang, T., Chen, Z., Su, Y., Feng, C., Yang, Z., & Lyu, M. R.-T. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00045 · Cited by 16
“To alleviate this issue, a complete and efficient pipeline should be constructed for monitoring data (e.g., monitoring metrics and logs [35], [36], [37]) collection and storage.”
BackgroundUsing approximate matching and machine learning to uncover malicious activity in logs
Flynn, R., & Olukoya, O. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104312 · Cited by 14
Zhou, J., Ying, S., Wang, S., Zhao, D., Xiang, J., Liang, K., & Liu, P. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3548050 · Cited by 17
“Unsupervised techniques like PCA [34] and LogCluster [35] grouped similar log entries to identify outliers.”
InfluentialAutomatic Problem Localization via Multi-dimensional Metric Profiling
Laguna, I., Mitra, S., Arshad, F. A., Theera-Ampornpunt, N., Zhu, Z., Bagchi, S., Midkiff, S. P., Kistler, M., & Gheith, A. (2013) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2013.21 · Cited by 14
“Some of the work analyzes applicatio n logs [18], [27], [28], [30].”
BackgroundSoftware Logs for Machine Learning in a DevOps Environment
Bosch, N., & Bosch, J. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00016 · Cited by 16
“There is existing research concerning the analysis of system logs including workflow [8], [17], process-mining [11], rule-based [3], PCA [14], [15], and LSTM language modelling [2], [4].”
MethodDomain Adaptive Log Anomaly Prediction for Hadoop System
Xie, Y., & Yang, K. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3178873 · Cited by 15
“Public Log Data Set [43]: The HDFS log data contains 11 175 629 log messages collected from the Amazon EC2 platform, of which 100 000 log messages are randomly selected to analyze in our experiments.”
MethodUtility Analysis about Log Data Anomaly Detection Based on Federated Learning
Shin, T.-H., & Kim, S.-H. (2023) · Applied Sciences · doi:10.3390/app13074495 · Cited by 18
Light-weight black-box failure detection for distributed systems
Tan, J., Kavulya, S. P., Gandhi, R., & Narasimhan, P. (2012) · doi:10.1145/2378356.2378360 · Cited by 14
“Some techniques have diagnosed failures using white-box information from Hadoop’s natively generated logs [14, 17, 21], rather than black-box OS-level metrics that LFD uses.”
MethodOtus: resource attribution in data-intensive clusters
Ren, K., López, J., & Gibson, G. A. (2011) · doi:10.1145/1996092.1996094 · Cited by 14
A Lightweight Tool for Anomaly Detection in Cloud Data Centres
Barbhuiya, S., Papazachos, Z. C., Kilpatrick, P., & Nikolopoulos, D. S. (2015) · International Conference on Cloud Computing and Services Science · doi:10.5220/0005453403430351 · Cited by 14
“Xu et al. (Xu et al., 2009) propose a new methodology to mine console logs to automatically detect system problems.”
BackgroundMethodInfluentialA Systematic Approach for Privilege Escalation Prevention
Jaafar, F., Nicolescu, G., & Richard, C. (2016) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2016.17 · Cited by 14
“In fact, the set of log messages of a one system is too huge to examine manually and too unstructured to analyze automatically [6].”
BackgroundHow Is Logging Practice Implemented in Open Source Software Projects? A Preliminary Exploration
Rong, G., Gu, S., Zhang, H., Shao, D., & Liu, W. (2018) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2018.00031 · Cited by 14
“Log analysis is also widely used in anomaly detection [1]–[3], [5], monitoring [4], cause diagnosis [6], [7], etc.”
BackgroundMethodDLME: Distributed Log Mining Using Ensemble Learning for Fault Prediction
Pal, A., & Kumar, M. (2019) · IEEE Systems Journal · doi:10.1109/jsyst.2019.2904513 · Cited by 14
“[23] carried out the parsing of variable and constant parts of the log messages and applied PCA for anomaly detection after converting the parsed logs into frequency domain.”
MethodCAT: content-aware tracing and analysis for distributed systems
Esteves, T., Neves, F., Oliveira, R., & Paulo, J. (2021) · International Middleware Conference · doi:10.1145/3464298.3493396 · Cited by 14
Live Forensics for HPC Systems: A Case Study on Distributed Storage Systems
Jha, S., Cui, S., Banerjee, S. S., Xu, T., Enos, J. J., Showerman, M., Kalbarczyk, Z., & Iyer, R. K. (2020) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41405.2020.00069 · Cited by 22
“, [32], [33]), (ii) prior work supports either (a) anomaly detection [34], [35], (b) failure localization [26], [27], [29]–[31], or (c) failure diagnosis only [36], [37].”
BackgroundEnd-to-End AutoML for Unsupervised Log Anomaly Detection
Zhang, S., Ji, Y., Luan, J., Nie, X., Chen, Z., Ma, M., Sun, Y., & Pei, D. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695535 · Cited by 31
Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection
Lim, Y. F., Zhu, J., & Pang, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8298-0_26 · Cited by 13
Unleashing the True Potential of Semantic-Based Log Parsing with Pre-Trained Language Models
Le, V.-H., Xiao, Y., & Zhang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00174 · Cited by 22
Scalable analysis of network measurements with Hadoop and Pig
Samak, T., Gunter, D., & Hendrix, V. (2012) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2012.6212060 · Cited by 13
Doomsday: predicting which node will fail when on supercomputers
Das, A., Mueller, F., Hargrove, P., Roman, E., & Baden, S. B. (2018) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.5555/3291656.3291668 · Cited by 13
Log Anomaly Detection Using Adaptive Universal Transformer
Wibisono, S. R., & Kistijantoro, A. I. (2019) · International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA) · doi:10.1109/icaicta.2019.8904299 · Cited by 14
Reconstruction-based anomaly detection for the cloud: A comparison on the Yahoo! Webscope S5 dataset
Hagemann, T., & Katsarou, K. (2020) · International Conference on Cloud and Big Data Computing (ICCBDC) · doi:10.1145/3416921.3416934 · Cited by 13
“A related approach from the area of anomaly detection in log files is for example [19].”
Ding, H., Yan, S., Zhai, J., & Ma, S. (2021) · USENIX Security Symposium · Cited by 37
Unsupervised Log Anomaly Detection Method Based on Multi-Feature
He, S., Deng, T., Chen, B., Sherratt, R. S., & Wang, J. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.037392 · Cited by 16
“A brief introduction to the baseline method is as follows: PCA [33] is a Principal Component Analysis (PCA)-based approach that extracts major components of input sequence features and reports test sequences above the threshold as anomalies.”
LogLead - Fast and Integrated Log Loader, Enhancer, and Anomaly Detector
Mäntylä, M., Wang, Y., & Nyyssölä, J. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00046 · Cited by 21
“For instance, in HDFS [7], sequences are created using the block id present in log message of each event.”
MethodImproving Log-Based Fault Diagnosis by Log Classification
Zou, D., Qin, H., Jin, H., Qiang, W., Han, Z., & Chen, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44917-2_37 · Cited by 13
“The important techniques of previous work are mainly relied on regular expressions [5].”
BackgroundMethodFmeter: Extracting Indexable Low-Level System Signatures by Counting Kernel Function Calls
Marian, T. S., Weatherspoon, H., Lee, K.-S., & Sagar, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35170-9_5 · Cited by 13
GRETEL: Lightweight Fault Localization for OpenStack
Goel, A., Kalra, S., & Dhawan, M. (2016) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/2999572.2999600 · Cited by 13
“Tracing systems [24, 35, 46, 48, 49] leverage log analysis to determine causal relationships among events.”
BackgroundPerformance analysis of idle programs
Altman, E., Arnold, M., Fink, S. J., & Mitchell, N. (2010) · ACM SIGPLAN Notices · doi:10.1145/1932682.1869519 · Cited by 13
CLDTLog: System Log Anomaly Detection Method Based on Contrastive Learning and Dual Objective Tasks
Tian, G., Luktarhan, N., Wu, H., & Shi, Z. (2023) · Sensors · doi:10.3390/s23115042 · Cited by 15
“The HDFS (Hadoop Distributed File System) [42] dataset generated by over 200 Amazon EC2 nodes contains 11,175,629 log messages, which form different log windows (log blocks) based on their block_id, reflecting the execution of a program in the HDFS system.”
MethodSpikeLog: Log-Based Anomaly Detection via Potential-Assisted Spiking Neuron Network
Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., & Qian, D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3347695 · Cited by 15
“In our experiments, we select four public log datasets, namely HDFS [28], BGL [29], Spirit [30], and Thunderbird [29].”
MethodLogETA: Time-aware cross-system log-based anomaly detection with inter-class boundary optimization
Gong, K., Luo, S., Pan, L., Zhang, L., Zhang, Y., & Yu, H. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.03.028 · Cited by 14
Interpretable Spatial–Temporal Graph Convolutional Network for System Log Anomaly Detection
Xu, R., & Li, Y. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102803 · Cited by 14
Zhang, L., Jia, T., Jia, M., Liu, H., Yang, Y., Wu, Z., & Li, Y. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3521675 · Cited by 20
“This is because Monitoring Data and Application Logs focus on diagnosing different types of anomalies [40], [41], while Raft Logs can identify a broader range of anomaly categories.”
SemiSMAC: A semi-supervised framework for log anomaly detection with automated hyperparameter tuning
Sun, Y., Keung, J., Yang, Z., Liu, S., & Liao, Y. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107869 · Cited by 13
Deep anomaly detection of temporal heterogeneous data in AIOps: a survey
Gui, J., Ma, Z., Zhou, H., Su, Y., Zhang, M., Yu, K., & Wu, X. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2400467 · Cited by 12
“Additionally, a privacy-preserving federated PCA framework has been devised, combining federated learning with PCA to extract traffic features from various IoT devices without compromising data exchange (Nguyen et al., 2023).”
Delog: A High-Performance Privacy Preserving Log Filtering Framework
Agrawal, A., Dixit, A., Shettar, N. A., Kapadia, D., Agrawal, V., Gupta, R. K., & Karlupia, R. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006218 · Cited by 15
“Six of these datasets comprise of distributed system logs, namely: Zookeeper [15], HDFS [16], HDFS2 [6], Spark,”
BackgroundSentinel: universal analysis and insight for data systems
Glasbergen, B., Abebe, M., Daudjee, K., & Levi, A. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3407790.3407856 · Cited by 12
“While other analysis tools generalize across systems by exploiting outputted debug logs to compare and contrast system behaviour [12, 14, 24, 34], the overheads of fine-grained debug logging are typically prohibitive [35,37].”
BackgroundIdentifying Anomaly Detection Patterns from Log Files: A Dynamic Approach
Cavallaro, C., & Ronchieri, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86960-1_36 · Cited by 15
Drill: Log-based Anomaly Detection for Large-scale Storage Systems Using Source Code Analysis
Zhang, D., Egersdoerfer, C., Mahmud, T., Zheng, M., & Dai, D. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00028 · Cited by 15
“As a result, an extensive amount of research on log-based anomaly detection tools has been conducted recently [4], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
BackgroundMethodInfluentialSpeed and Performance of Parserless and Unsupervised Anomaly Detection Methods on Software Logs
Nyyssölä, J., & Mäntylä, M. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00071 · Cited by 14
“The datasets used in the study are BGL [21], Thunderbird [21], HDFS [22] and Hadoop [23], which are easily accessible via LogHub [24].”
MethodAn Approach to Pinpointing Bug-Induced Failure in Logs of Open Cloud Platforms
Tong, J., Li, Y., Hongyan, T., & Wu, Z. (2016) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2016.0047 · Cited by 12
IT troubleshooting with drift analysis in the DevOps era
Meng, F. J., Wegman, M. N., Xu, J. M., Zhang, X., Chen, P. B., & Chafle, G. B. (2017) · IBM Journal of Research and Development · doi:10.1147/jrd.2016.2630478 · Cited by 15
Ranking Anomalous High Performance Computing Sensor Data Using Unsupervised Clustering
Morrow, A., Baseman, E., & Blanchard, S. P. (2016) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci.2016.0124 · Cited by 12
AECID-PG: A Tree-Based Log Parser Generator To Enable Log Analysis
Wurzenberger, M., Landauer, M., Skopik, F., & Kästner, W. (2019) · Immunotechnology · Cited by 12
Chuah, E., Jhumka, A., Alt, S., Balouek‐Thomert, D., Browne, J. C., & Parashar, M. (2019) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2019.05.013 · Cited by 12
Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism
Gu, S., Chu, Y., Zhang, W., Liu, P., Yin, Q., & Li, Q. (2021) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd51990.2021.9459087 · Cited by 12
ConfInLog: Leveraging Software Logs to Infer Configuration Constraints
Zhou, S., Liu, X., Li, S., Jia, Z., Zhang, Y., Wang, T., Li, W., & Liao, X. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00018 · Cited by 12
LogDP: Combining Dependency and Proximity for Log-Based Anomaly Detection
Xie, Y., Zhang, H., Zhang, B., Babar, M. A., & Lu, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91431-8_47 · Cited by 16
Anomaly Detection in Unstructured Logs Using Attention-based Bi-LSTM Network
Yu, D., Hou, X., Li, C., Lv, Q., Wang, Y., & Li, N. (2021) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc54101.2021.9660476 · Cited by 12
Log Anomaly to Resolution: AI Based Proactive Incident Remediation
Mahindru, R., Kumar, H., & Bansal, S. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678815 · Cited by 21
“One of the initial steps in detecting log anomalies is parsing the logs to generate the templates [3], and the different methods include PCA [4], Invariant mining [5], [6], Log Clustering [7], [8], Neural Granger Causality [9] and Deep Learning based approaches [10]–[15].”
MethodQuickLogS: A Quick Log Parsing Algorithm based on Template Similarity
Fang, L., Di, X., Liu, X., Qin, Y., Ren, W., & Ding, Q. (2021) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom53373.2021.00148 · Cited by 14
“The six datasets in the table are all public, HDFS logs are collected in [35] by engaging a 203-node cluster on Amazon EC2 platform, with 11,175,629 log messages.”
MethodTowards Detecting Patterns in Failure Logs of Large-Scale Distributed Systems
Gurumdimma, N. Y., Jhumka, A., Liakata, M., Chuah, E., & Browne, J. C. (2015) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2015.109 · Cited by 12
“[27] and [22] used console logs combine with source code to create useful features that can be used by analysis algorithms for detecting faulty events.”
MethodToward Autonomic Cloud: Automatic Anomaly Detection and Resolution
Ahad, R., Chan, E., & dos Santos, A. (2015) · International Conference on Cloud and Autonomic Computing (ICCAC) · doi:10.1109/iccac.2015.32 · Cited by 12
“principal component analysis to reduce the metric dimension [8], and by a combination of program analysis and information retrieval techniques to transform free-text console logs to numerical features [9].”
MethodHytrace: A Hybrid Approach to Performance Bug Diagnosis in Production Cloud Infrastructures
Dai, T., Dean, D. J., Wang, P., Gu, X., & Lu, S. (2018) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2018.2858800 · Cited by 12
Improvement of the Log Pattern Extracting Algorithm Using Text Similarity
Zhao, Y., Wang, X., Xiao, H., & Chi, X. (2018) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2018.00087 · Cited by 12
ECLogger: Cross-Project Catch-Block Logging Prediction Using Ensemble of Classifiers
Lal, S., Sardana, N., & Sureka, A. (2016) · Prace Naukowe Uniwersytetu Ekonomicznego we Wrocławiu · doi:10.5277/e-inf170101 · Cited by 17
“[18] worked on mining console logs from a distributed system at Google to find anomalies in the system.”
BackgroundThe Statistics of Streaming Sparse RegressionPreprint
Steinhardt, J., Wager, S., & Liang, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4182 · Cited by 12
“In many areas such as astrophysics [1, 6], environmental sensor networks [42], distributed computer systems diagnostics [61], and advertisement click prediction [36], a system generates a high-throughput stream of data in real-time.”
BackgroundContemporary Software Monitoring: A Systematic Literature Review.Preprint
Cândido, J., Aniche, M. F., & van Deursen, A. (2019) · arXiv (Cornell University) · Cited by 13
DyCause: Crowdsourcing to Diagnose Microservice Kernel Failure
Pan, Y., Ma, M., Jiang, X., & Wang, P. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3233915 · Cited by 11
“Most of them first model the microservice system as static structures such as a hybrid graph and Markov models and then detect problems from the deviations of previous structures [8], [9], [10], [11].”
BackgroundFrom Logs to Causal Inference: Diagnosing Large Systems
Markakis, M., Youngmann, B., Gao, T., Zhang, Z., Shahout, R., Chen, P. B., Liu, C., Sabek, I., & Cafarella, M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3705829.3705836 · Cited by 14
Automatic Procedure Following Evaluation Using Petri Net-Based Workflows
Rodríguez-Fernández, V., González-Pardo, A., & Camacho, D. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2779177 · Cited by 11
“of a data log comes from both the ideas presented in [17], where authors convert unstructured text into a structured log that shows the logged variables in (name, value) pairs, and the notation used for Data PNs in [18], to which we have added the notions of temporal dependency in the values of variables.”
MethodUnsupervised Signature Extraction from Forensic Logs
Thaler, S., Menkovski, V., & Petković, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71273-4_25 · Cited by 11
“Instead, they propose to create to extract the signatures from the source code [26].”
BackgroundNowhere to Hide Methodology: Application of Clustering Fault Diagnosis in the Nuclear Power Industry
Zong, C., Huang, S., Liu, E., Yao, Y., & Tang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2957807 · Cited by 11
“Sherlog [4], ElasticSearch [28], SmartLog [31], Statechart Workbench [32] and other methods [6], [20], [26] have been designed for fault diagnosis and troubleshooting; however, the above tools and approaches make a common assumption that the source code of the system is available.”
BackgroundMethodInfluentialLogging statements' prediction based on source code clones
Gholamian, S., & Ward, P. A. S. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3373845 · Cited by 11
“For example, many studies have applied the content of log files to achieve a variety of goals such as anomaly and problem detection [5, 25], pattern detection and log message clustering [14, 24], system profile building, and compression of log files [14, 22].”
Geepalla, E., & Asharif, S. (2020) · International Conference on Engineering & MIS (ICEMIS) · doi:10.1145/3410352.3410817 · Cited by 11
“In such a setting, the traditional method using keyword search becomes ineffective for extracting suspicious log messages in these systems, which likely leads to many false positives that are actually log messages unrelated to real failures [11, 12, 13,].”
MethodA Security Log Analysis Scheme Using Deep Learning Algorithm for IDSs in Social Network
Zhong, M. J., Zhou, Y., & Chen, G. (2021) · Security and Communication Networks · doi:10.1155/2021/5542543 · Cited by 11
“)e HDFS data contains 11,175,629 log messages, which are Hadoop logs collected from the Amazon EC2 platform, and are marked as normal or abnormal by experts in the relevant field [36].”
MethodPinpointing Anomaly Events in Logs from Stability Testing – N-Grams vs. Deep-Learning
Mäntylä, M., Varela, M., & Hashemi, S. (2022) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw55395.2022.00056 · Cited by 16
“To contrast the long log sequence of the case company (45,474 events on average), we use open data from HDFS [11] with 10,9 million log events from 588 thousand log sequences that have an average of 18.”
BackgroundMethodSoftware Failure Log Analysis for Engineers—Review
Dobrowolski, W., Nikodem, M., & Unold, O. (2023) · Electronics · doi:10.3390/electronics12102260 · Cited by 12
Hilogx: noise-aware log-based anomaly detection with human feedback
Jia, T., Li, Y., Yang, Y., & Huang, G. (2024) · The VLDB Journal · doi:10.1007/s00778-024-00843-2 · Cited by 14
“…detection models can be broadly classified into supervised models [7, 24, 28, 35, 41] and unsupervised models [6, 11, 13, 14, 17, 18, 20, 22, 27, 31–34 , 37– 40 Unsupervised models avoid the limitations of supervised models by only using the normal log sequences for training and they are more…”
Self-Evolutionary Group-wise Log Parsing Based on Large Language Model
Pei, C., Liu, Z., Li, J., Zhang, E., Zhang, L., Zhang, H., Chen, W., Pei, D., & Xie, G. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00016 · Cited by 20
“The earliest log parser directly parsed the system code and extracted the log template according to the log output statement [23], [24] which is shown in the source code part of Fig.”
MapReduce based log file analysis for system threats and problem identification
Vernekar, S., & Buchade, A. (2013) · IEEE International Advance Computing Conference · doi:10.1109/iadcc.2013.6514334 · Cited by 11
“The paper [15] provides a methodology to mine rich source of information from console logs to automatically detect system runtime problems.”
MethodAn automatic approach to extract the formats of network and security log messages
Ya, J., Liu, T., Zhang, H., Shi, J., & Guo, L. (2015) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom.2015.7357664 · Cited by 11
“Much similar format extracting work has been done in many different research fields [5], [6], [10], [11], [12], [13], [14], [15], [7], [16], [17], which can be classified into two categories: reverse engineering and automatic inference.”
MethodUnderstanding and Statically Detecting Synchronization Performance Bugs in Distributed Cloud Systems
Zhang, C., Li, J., Li, D., & Lu, X. (2019) · IEEE Access · doi:10.1109/access.2019.2923956 · Cited by 11
An Attention-Based GRU Network for Anomaly Detection from System Logs
Xie, Y.-X., Ji, L., & CHENG, X. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2020edl8016 · Cited by 12
“We compare our approach against following widely applied methods, which are PCA (principal component analysis) [2], IM (invariant mining) [3], DeepLog [5], GRU-only and LogGAN [10].”
MethodMassive data clustering by multi-scale psychological observations
Yang, S., Zhang, L., Xu, C., Yu, H., Fan, J., & Xu, Z. (2021) · National Science Review · doi:10.1093/nsr/nwab183 · Cited by 11
“This experiment is based on the SOSP 2009 dataset [37] containing11 197 705 logsof theHDFS[38] in a private cloud.”
MethodA comparison of syslog and IS-IS for network failure analysis
Turner, D., Levchenko, K., Savage, S., & Snoeren, A. C. (2013) · ACM Internet Measurement Conference (IMC) · doi:10.1145/2504730.2504766 · Cited by 11
“parse syslog records to identify anomalies in data center operations [20].”
BackgroundUncovering JavaScript Performance Code Smells Relevant to Type Mutations
Xiao, X., Shi, H., Zhang, C., & Zhang, D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-26529-2_18 · Cited by 11
“Most of the works still rely on function execution time profiling data and statistical algorithms [20, 23, 18, 25, 3, 24, 8, 13].”
MethodIt Can Understand the Logs, Literally
Pi, A., Chen, W., Zeller, W., & Zhou, X. (2019) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2019.00084 · Cited by 13
“For automatic anomaly detection, a number of studies apply machine learning methods to logs [4], [5], [6], [7], [8], [9], [10].”
MethodAAMR: Automated Anomalous Microservice Ranking in Cloud-Native Environment
Zhang, Z. (2021) · International Conferences on Software Engineering and Knowledge Engineering · doi:10.18293/seke2021-091 · Cited by 10
Log Anomaly Detection Method Based on Transformer and Temporal Convolutional Networks
Liao, N., & Liu, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3561669 · Cited by 13
“(1) Hadoop Distributed File System (HDFS) [11]: The dataset was generated by executing MapReduce tasks on 203 Amazon EC2 nodes.”
Insight: in-situ online service failure path inference in production computing infrastructures
Nguyen, H., Dean, D. J., Kc, K., & Gu, X. (2014) · Cited by 10
Detecting Anomalous Events on Distributed Systems Using Convolutional Neural Networks
Cheansunan, P., & Phunchongharn, P. (2019) · International Conference on Awareness Science and Technology (iCAST) · doi:10.1109/icawst.2019.8923357 · Cited by 10
“For our performance evaluation, we employed the HDFS log dataset in [10].”
MethodStudying and Complementing the Use of Identifiers in Logs
Zhao, J., Tang, Y., Sunil, S., & Shang, W. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00019 · Cited by 10
“In addition, software logs keep track of crucial run-time information that developers commonly use to facilitate software testing and development activities, including debugging [2], [3], [4], monitoring software processes [5] and transferring knowledge [6].”
MethodADAL-NN: Anomaly Detection and Localization Using Deep Relational Learning in Distributed Systems
Ahmed, K. M., Altaf, A., Jamail, N. S. M., Iqbal, F., & Latif, R. (2023) · Applied Sciences · doi:10.3390/app13127297 · Cited by 11
“The HDFS log collection was obtained from a cluster of 203 nodes on the Amazon EC2 platform [3], and it contains 11,176,738 raw log messages.”
BackgroundMethodInfluentialLogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation
Deng, L., Liu, S., Xu, H., & Wang, W. (2023) · Applied Sciences · doi:10.3390/app13137739 · Cited by 14
“Since PCA, LogCluster, DeepLog, LogAnomaly, and LogBERT are unsupervised models and are not designed for cross-system detection, these models were evaluated in two cases; that is, the training dataset used or did not use the samples of the target system, represented by W/O. Firstly, the Thunderbird dataset was used as…”
MethodInfluentialTowards a privacy-aware stream data management system for cloud applications
Ng, W. S., Kirchberg, M., Bressan, S., & Tan, K. (2011) · International Journal of Web and Grid Services · doi:10.1504/ijwgs.2011.043530 · Cited by 10
Dong, W., Chen, C., Bu, J., Liu, X., & Liu, Y. (2013) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss.2013.28 · Cited by 10
“Distalyzer [34] uses machine learning techniques to compare system behaviors extracted from the logs and automatically infer the strongest associations between system components and performance.”
MethodInfluentialBig Data and High Performance Computing
Chen, W. (2015) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.11959/j.issn.2096-0271.2015003 · Cited by 10
Holistic Processing and Exploring Event Logs
Kubacki, M., & Sosnowski, J. S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-65948-0_12 · Cited by 10
Log Parser with One-to-One Markup
Chunyong, Z., & Meng, X. (2020) · International Conference on Information and Computer Technologies (ICICT) · doi:10.1109/icict50521.2020.00045 · Cited by 10
“Recent literature confirms the important contribution of log analysis technology to other work, in anomaly detection[1][2],fault diagnosis[3][4],performance improvement[5][6], business model mining[7][8],and user behavior analysis[9][10], etc.”
BackgroundMethodInfluentialSynoptic: summarizing system logs with refinement
Schneider, S., Beschastnikh, I., Chernyak, S., Ernst, M. D., & Brun, Y. (2010) · doi:10.1145/1928991.1928995 · Cited by 15
“Prior work on distributed system log mining focused on detecting dependencies [25], anomalies [36, 19, 41], and performance debugging [32, 33].”
BackgroundStructured and Interoperable Logging for the Cloud Computing Era: The Pitfalls and Benefits
Tovarňák, D., Vaekova, A., Novak, S., & Pitner, T. (2013) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc.2013.30 · Cited by 10
“They must deal with a growing amount and heterogeneity of logs produced by cloud datacenters and therefore growing complexity and number of regular expressions to be tested (the worst case scenario is one regular expression for each existing log type).”
BackgroundTwo Level Empirical Study of Logging Statements in Open Source Java Projects
Lal, S., Sardana, N., & Sureka, A. (2015) · International Journal of Open Source Software and Processes · doi:10.4018/ijossp.2015010104 · Cited by 10
Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis
Chuah, E., Jhumka, A., Alt, S., Damoulas, T., Gurumdimma, N. Y., Sawley, M.-C., Barth, W. L., Minyard, T., & Browne, J. C. (2017) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc.2017.00044 · Cited by 10
“A significant body of research has shown the value of message logs for error detection [3]– [5], failure prediction [6]–[12] and failure diagnosis [13]–[15].”
BackgroundLeveraging Code Clones and Natural Language Processing for Log Statement Prediction
Gholamian, S. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678596 · Cited by 10
“This logging process enables developers and system administrators to analyze log files for a variety of purposes [1], such as anomaly and problem detection [2, 3], log message clustering [4, 5], system profile building [6], code quality assessment [7], and compression of log files [4, 8].”
Using program analysis to reduce misconfiguration in open source systems software
Katz, R. H., & Rabkin, A. (2012) · Cited by 10
Software Aging and Rejuvenation for Increased Resilience: Modeling, Analysis and Applications
Avritzer, A., Czekster, R. M., Distefano, S., & Trivedi, K. S. (2012) · Springer eBooks · doi:10.1007/978-3-642-29032-9_8 · Cited by 10
PATHA: Performance Analysis Tool for HPC Applications
Yoo, W., Koo, M. S., Cao, Y., Sim, A., Nugent, P., & Wu, K. J. (2015) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/pccc.2015.7410313 · Cited by 16
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.