Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 601–700 of 1,004
Mantis: Predicting System Performance through Program Analysis and ModelingPreprint
Chun, B.-G., Huang, L., Lee, S., Maniatis, P., & Naik, M. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.0019 · Cited by 10
“ource consumption or workload size for prediction. Xu et al. used console logs — coarse-grainedprogram status reports — to detect anomalies in the Hadoop Distributed File System and the Darkstar game [37]. For load shedding in network monitoring applications, Barlet-Ros et al. [10] used a simple linear regression mode…”
MethodSelf-citationGraph Neural Networks Based Log Anomaly Detection and ExplanationPreprint
Li, Z., Shi, J., & van Leeuwen, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4627217 · Cited by 19
Yicheng, S. U. N., Keung, J., Yang, Z., Liu, S., Zhang, J., & Liao, Y. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00548-y · Cited by 10
“We employed two public datasets to evaluate the performance of our model: HDFS (Xu et al. 2009 ), BGL (Oliner and Stearley 2007 ), Thunderbird (Oliner and Stearley 2007 ), and Spirit (Oliner and Stearley 2007 ) datasets.”
K 4 : Online Log Anomaly Detection Via Unsupervised Typicality Learning
Chen, W., Singh, V., Rahmani, Z., Ganguly, D., Hariri, M., & Chaudhary, V. (2025) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc66333.2025.00019 · Cited by 12
“In large-scale environments such as high-performance computing (HPC) systems, cloud infrastructures, and edge platforms, automated log anomaly detection (LogAD) is vital for prompt error recovery, root-cause diagnosis, and proactive system management [1]–[3].”
Failure Identification from Unstable Log Data using Deep Learning
Bogatinovski, J., Nedelkoski, S., Li, W., Cardoso, J., & Kao, O. (2022) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid54584.2022.00044 · Cited by 11
Bayesian Network analysis of software logs for data‐driven software maintenance
del Rey, S., Martínez‐Fernández, S., & Salmerón, A. (2023) · IET Software · doi:10.1049/sfw2.12121 · Cited by 9
Han, D., Sun, M., Li, M., & Chen, Q. (2023) · Applied Sciences · doi:10.3390/app13137668 · Cited by 10
FSMFLog: Discovering Anomalous Logs Combining Full Semantic Information and Multifeature Fusion
Niu, W., Li, Z., He, Z., Wang, A., Li, B., & Zhang, X. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3300690 · Cited by 11
“LogPai [12] collected 16 real-world system log data sets (HDFS [32], Hadoop, Spark, Zookeeper, BG [33], HPC, Thunderbird, Windows, Linux, Android, HealthApp, Apache, proxy, OpenSSH, OpenStack, and Mac) stored in loghub [34] and evaluated 13 different log parsers in terms of accuracy, robustness,…”
MethodDeep Learning-Based Log Anomaly Detection for 5G Core Network
Tan, Y., Wang, J., Liu, J., & Li, Y. (2023) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/iccc57788.2023.10233555 · Cited by 12
Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder
Zhang, X., Chai, X., Yu, M., & Qiu, D. (2023) · International Conference on Information Science, Parallel and Distributed Systems (ISPDS) · doi:10.1109/ispds58840.2023.10235370 · Cited by 9
“[9] used PCA method for anomaly detection.”
Fuzzy CNN Autoencoder for Unsupervised Anomaly Detection in Log Data
Gorohov, O., Petrovskiy, M., Машечкин, И. В., & Kazachuk, M. (2023) · Mathematics · doi:10.3390/math11183995 · Cited by 10
Zhao, X., Jia, T., He, M., Wu, Y., Li, Y., & Huang, G. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728519 · Cited by 9
“We conducted experiments on three publicly available log datasets: HDFS [28], BGL [22] and OpenStack [3].”
Time Capsule: Tracing Packet Latency across Different Layers in Virtualized Systems
Suo, K., Rao, J., Cheng, L., & Lau, F. C. M. (2016) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/2967360.2967376 · Cited by 9
Log Analytics in HPC: A Data-driven Reinforcement Learning Framework
Luo, Z., Hou, T., Nguyen, T. T., Zeng, H., & Lu, Z. (2020) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps50562.2020.9162664 · Cited by 9
“Source code are also leveraged in [18] to get structured files.”
BackgroundAnomaly Detection Using System Logs: A Deep Learning Approach
Sinha, R., Sur, R., Sharma, R., & Shrivastava, A. K. (2021) · International Journal of Information Security and Privacy · doi:10.4018/ijisp.285584 · Cited by 11
Efficient Online Log Parsing with Log Punctuations Signature
Zhang, S., & Wu, G. (2021) · Applied Sciences · doi:10.3390/app112411974 · Cited by 9
SIERRA: Ranking Anomalous Activities in Enterprise Networks
Lee, J.-H., Tang, F., Thet, P. M., Yeoh, D., Rybczynski, M., & Divakaran, D. M. (2022) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp53844.2022.00011 · Cited by 13
“The early systems modeling log-sequences [34] and log counts of individual log type [35] require a fixed set of log types to define the log sequences and count the same type of logs; whereas a SIEM system has multiple MBs and periodically updates the logging policies.”
BackgroundAugLog: System Log Anomaly Detection Based on Contrastive Learning and Data Augmentation
Zhou, J., & Qian, Y. (2022) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit55514.2022.9943918 · Cited by 9
“For example, approaches based on machine learning including: decision tree [4], principal component analysis (PCA) [5], [6], support vector machine (SVM) [7], [8], and invariant mining (IM) [9].”
MethodInfluentialML-Parser: An Efficient and Accurate Online Log Parser
Zhu, Y.-Q., Deng, J.-Y., Pu, J.-C., Wang, P., Shen, L., & Wang, W. (2022) · Journal of Computer Science and Technology · doi:10.1007/s11390-021-0730-4 · Cited by 9
“To avoid tedious manual intervention, Xu et al. [6] proposed an approach using source code knowledge to infer message templates.”
MethodLogBASA: Log Anomaly Detection Based on System Behavior Analysis and Global Semantic Awareness
Liao, L., Zhu, K., Luo, J., & Cai, J. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/3777826 · Cited by 10
Hound: Causal Learning for Datacenter-scale Straggler Diagnosis
Zheng, P., & Lee, B. C. (2018) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/3219617.3219641 · Cited by 9
Winter, J., Aniche, M. F., Cito, J., & Arie Van Deursen, A. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338926 · Cited by 9
“’s heuristic [61] to link the log line back to its originating log statement.”
BackgroundMethodA System Fault Diagnosis Method with a Reclustering Algorithm
Yang, Z., Ying, S., Wang, B., Li, Y., Dong, B., Geng, J., & Zhang, T. (2021) · Scientific Programming · doi:10.1155/2021/6617882 · Cited by 9
On the Naturalness and Localness of Software Logs
Gholamian, S., & Ward, P. A. S. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00028 · Cited by 14
“research [20], [28], [29], [37], [48], [49], [65].”
BackgroundMethodResultInfluentialMithriLog: Near-Storage Accelerator for High-Performance Log Analytics
Kang, S., An, J., Kim, J.-P., & Jun, S.-W. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480108 · Cited by 11
“Collecting and analyzing log data in both cloud and edge deployments is a critical tool for large-scale system management including monitoring and insight [4, 47, 52, 82], as well as detecting abnormal behavior and security issues [12, 18, 34, 63, 79], and even program verification [1].”
BackgroundMethodKobayashi, S., Yamashiro, Y., Otomo, K., & Fukuda, K. (2021) · International Journal of Network Management · doi:10.1002/nem.2195 · Cited by 9
“There are some literature that use log templates extracted from source codes.(18,19,56) These templates are not parsed as the sequence of words, and cannot be segmented directly because the variable part can include separator symbols that are not visible in the templates.”
MethodLogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems
Sun, L., & Xu, X. (2023) · Security and Communication Networks · doi:10.1155/2023/2803139 · Cited by 10
System State Discovery Via Information Content Clustering of System Logs
Makanju, A., Zincir-Heywood, N., & Milios, E. (2011) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1109/ares.2011.51 · Cited by 9
“An obstacle to this investigation is the fact that clocks among several computers are usually not well synchronized to the level of precision that is required for such analysis [5], [16].”
BackgroundA query language for understanding component interactions in production systems
Oliner, A. J., & Aiken, A. (2010) · International Conference on Supercomputing · doi:10.1145/1810085.1810114 · Cited by 9
“Recent work shows how access to source code can facilitate tasks like log analysis [30] and distributed diagnosis [12].”
BackgroundDesign and Implementation of a Semi-supervised Anomaly Log Detection Model DDA
Wang, Y., & Ji, Z. (2021) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai50917.2021.9447533 · Cited by 9
Log anomaly detection based on BERT
Tang, P., & Guan, Y. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03327-6 · Cited by 9
Efficient Log-based Anomaly Detection with Knowledge Distillation
Nguyen, H., Nguyen, L.-V., Le, V.-H., Zhang, H., & Le, M.-T. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00078 · Cited by 12
“Many approaches have been proposed to apply ML [8], [24] and DL [3], [9], [59] methods for automated log-based anomaly detection, an attractive topic in both academia and industry.”
MethodOMLog: Online Log Anomaly Detection for Evolving System With Meta-Learning
Tian, J., Li, M., Wang, Z., Chen, L., Qin, J., & Zhang, R. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3569628 · Cited by 10
“The HDFS dataset [27] consists of 11,175,629 logs produced from more than 200 Amazon EC2 nodes spanning 38.7 h.”
InfluentialRhee, M., Zou, J., & Qin, Y. (2024) · AI and Data Science Journal · doi:10.61784/adsj4039 · Cited by 25
“Logs are attractive because they encode control-flow and component-state evidence unavailable in aggregate metrics; however, they are noisy, duplicated, imbalanced, and affected by parser evolution [4-9].”
Interactive debugging for big data analytics
Gulzar, M. A., Han, X., Interlandi, M., Mardani, S., Tetali, S. D., Condie, T., Millstein, T. D., & Kim, M. (2016) · IEEE International Conference on Cloud Computing Technology and Science · Cited by 8
REVAL: Recommend Which Variables to Log With Pretrained Model and Graph Neural Network
Dai, S., Luan, Z., Huang, S., Fung, C., Wang, H., Yang, H., & Qian, D. (2022) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3209317 · Cited by 10
“…information that record software run-time status, bug elimination, QoS (Quality of Service) improvements, and many assurance tasks such as anomaly detection [4], [5], fault diagnosis [6], [7], deployment(program) verification [8], [9], and performance monitoring [10], [11] are based on logs.”
BackgroundSawmill: From Logs to Causal Diagnosis of Large Systems
Markakis, M., Chen, A. B., Youngmann, B., Gao, T., Zhang, Z., Shahout, R., Chen, P. B., Liu, C., Sabek, I., & Cafarella, M. (2024) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3626246.3654731 · Cited by 11
“Users can optionally pass into Parse regular expressions for variables like timestamps.”
BackgroundAlaca, Y., Başaran, E., & Çelik, Y. (2024) · ADBA computer science. · doi:10.69882/adba.cs.2024073 · Cited by 8
Intent-Driven Multi-Engine Observability Dataflows for Heterogeneous Geo-Distributed Clouds
Chakraborty, A., Eswaran, A., Thorat, P. B., Verma, M., Gupta, P., & Jayachandran, P. (2024) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud62652.2024.00014 · Cited by 8
“Several approaches have been proposed in the literature for problem identification or root cause analysis based on machine learning [21], [22], static analysis [23], [24], causal analysis [25]–[27], and extracting execution models [28], [29].”
MethodLogGenius: An Unsupervised Log Parsing Framework with Zero-shot Prompt Engineering
Yu, X., Nong, S., He, D., Zheng, W., Ma, T., Liu, N., Li, J., & Xie, G. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00159 · Cited by 13
“These methods extract variables by parsing the corresponding source code based on abstract syntax trees (AST) [12].”
MethodTCMS: A Multi-Sequence Log Parsing Method Based on Token Conversion
Wei, M., Wen, J., He, S., Xie, K., Liang, W., Xie, G., Li, K., & Zhu, Z. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3520628 · Cited by 8
PADM: Page Rank-Based Anomaly Detection Method of Log Sequences by Graph Computing
Yan, X., Zhou, W. J., Gao, Y., Zhang, Z., Han, J., & Fu, G. Y. (2014) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2014.70 · Cited by 8
“For this reason, log-based post-found anomaly detection draws more and more attentions from both the academic and industrial fields[2][3].”
BackgroundOn the use of weighted syslog time series for anomaly detection
Fukuda, K. (2011) · IFIP/IEEE International Symposium on Integrated Network Management (IM) · doi:10.1109/inm.2011.5990538 · Cited by 8
“[22] proposed a template mechanism of the message types automatically generated from the source code of the target system.”
BackgroundMethodInfluentialCommunication-efficient Outlier Detection for Scale-out Systems.
Gabel, M., Keren, D., & Schuster, A. (2013) · Cited by 8
Debugging OpenStack Problems Using a State Graph Approach
Xiang, Y., Li, H., Wang, S., Chen, C. P., & Xu, W. (2016) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/2967360.2967366 · Cited by 8
“The authors of [7] detect problems using common identifiers in text logs.”
BackgroundMethodInfluentialSelf-citationPoSBert: Log Classification via Modified Bert Based on Part-of-Speech Weight
Zhang, J., Li, Z., Zhang, X., Lin, F., Wang, C., & Cai, X. (2022) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai55851.2022.9904207 · Cited by 8
“Compared with the public dataset, the practical dataset is of more significance to prove the improvement of our model.”
BackgroundLogformer: Cascaded Transformer for System Log Anomaly Detection
Hang, F., Guo, W., Chen, H., Xie, L., Zhou, C., & Liu, Y. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.025774 · Cited by 11
LADDERS: Log Based Anomaly Detection and Diagnosis for Enterprise Systems
Mondal, S. A., Rv, P., Rao, S., & Menon, A. (2023) · Annals of Data Science · doi:10.1007/s40745-023-00471-7 · Cited by 8
KVSAgg: Secure Aggregation of Distributed Key-Value Sets
Wu, Y., Dong, S., Zhou, Y., Zhao, Y., Fu, F., Yang, T., Niu, C., Wu, F., & Cui, B. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00139 · Cited by 8
“The local states can be many KV pairs of the event/resource identifier and the frequency (or any additive value like the web-page visiting time) [34], [35].”
BackgroundTransparent System Call Based Performance Debugging for Cloud Computing
Khadke, N., Kasick, M. P., Kavulya, S. P., Tan, J., & Narasimhan, P. (2012) · Cited by 8
Tell-tale tails: Decomposing response times for live internet services
Deng, N., Xu, Z., Stewart, C., & Wang, X. (2015) · International Green and Sustainable Computing Conference · doi:10.1109/igcc.2015.7393717 · Cited by 8
“[24] use static analysis to find log statements in the source code and relate the anomalous log messages back to the source code.”
Priolog: Mining Important Logs via Temporal Analysis and Prioritization
Tak, B., Park, S., & Kudva, P. N. (2019) · Sustainability · doi:10.3390/su11226306 · Cited by 9
“[30] proposed a technique of automatically creating features and applying PCA to detect anomalies.”
MethodLogSpy: System Log Anomaly Detection for Distributed Systems
Li, H., & Li, Y. (2020) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice51518.2020.00073 · Cited by 12
“[4] used Principal Component Analysis (PCA) and AST Abstract Syntax Tree (AST) methods to process the parsed log feature set, which reduced the complexity of the feature set to be analyzed and obtained effective the results of anomaly detection.”
MethodAnomaly detection on software log based on Temporal Memory
Hirakawa, R., Uchida, H., Nakano, A., Tominaga, K., & Nakatoh, Y. (2021) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2021.107433 · Cited by 9
System Failure Detection Using Deep Learning Models Integrating Timestamps With Nonuniform Intervals
Huangfu, Y., Habibi, S. R., & Wassyng, A. (2022) · IEEE Access · doi:10.1109/access.2022.3150342 · Cited by 12
“Learning models can be formulated in a supervised manner, such as the logistic regression [18], decision tree [2], and support vector machines [3], [11], [21], as well as in an unsupervised manner, such as principal component analysis [14], invariant mining [17], and K-means clustering [24].”
BackgroundMethodInfluentialLogDAC: A Universal Efficient Parser-based Log Compression Approach
Tian, R., Diao, Z., Jiang, H., & Xie, G. (2022) · International Conference on Communications · doi:10.1109/icc45855.2022.9838258 · Cited by 8
“Systems generate logs to store critical information for system diagnostics [1], anomaly detection [2, 3, 4, 5], user behavior prediction [6] and event root cause analysis [7, 8], etc. Newly developed techniques such as 5G, Big 5G, or Beyond 5G cloud core network and its User Plane Functions also…”
MethodHow to Configure Masked Event Anomaly Detection on Software Logs?
Nyyssölä, J., Mäntylä, M., & Varela, M. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00050 · Cited by 8
ASDF: An Automated, Online Framework for Diagnosing Performance Problems
Bare, K. A., Kavulya, S. P., Tan, J., Pan, X., Marinelli, E., Kasick, M. P., Gandhi, R., & Narasimhan, P. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17245-8_9 · Cited by 8
“al [23] diagnosed problems in Hadoop by analyzing source code to automatically discover the structure of messages in the datanode, logs, and identifying outlier error messages.”
MethodDetecting performance anomaly with correlation analysis for Internetware
Wang, T., Wei, J., Qin, F., Zhang, W., Zhong, H., & Huang, T. (2013) · Science China Information Sciences · doi:10.1007/s11432-013-4906-6 · Cited by 8
amulog: A General Log Analysis Framework for Diverse Template Generation Methods
Kobayashi, S., Yamashiro, Y., Otomo, K., & Fukuda, K. (2020) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm50824.2020.9269049 · Cited by 8
“For example, a set of generated templates from source code analysis [15]–[17] is not mapped to the real log messages (i.e., does not solve the classification issue).”
BackgroundMethodEnCore: exploiting system environment and correlation information for misconfiguration detection
Zhang, J., Renganarayana, L., Zhang, X., Ge, N., Bala, V., Xu, T., & Zhou, Y. (2014) · ACM SIGARCH Computer Architecture News · doi:10.1145/2654822.2541983 · Cited by 8
LogBERT: Log Anomaly Detection via BERT
Guo, H., Yuan, S., & Wu, X. (2021) · arXiv · doi:10.48550/arxiv.2103.04475 · Cited by 7
“Regarding baselines, we leverage the package Loglizer [4] to evaluate PCA, OCSVM, iForest as well as LogCluster for anomaly detection and adopt the open source deep learning-based log analysis toolkit to evaluate DeepLog and LogAnomaly 3 .”
InfluentialLandscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods
Lupton, S., Washizaki, H., Yoshioka, N., & Fukazawa, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3387287 · Cited by 9
“Offline parsers extract templates either by directly refer-encing log output statements from system source code [8] or by deriving templates from historic log data through algorithmic means [9], [10], [11].”
MethodPluto: Sample Selection for Robust Anomaly Detection on Polluted Log Data
Ma, L., Cao, L., VanNostrand, P. M., Hofmann, D. M., Su, Y., & Rundensteiner, E. A. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3677139 · Cited by 10
“• Hadoop Distributed File System [46] (HDFS) is a log dataset collected from a Hadoop-based map-reduce cloud environment using benchmark queries.”
Anomaly Transformer-based System Log Anomaly Detection
Shi, Q., Yang, Z., Haidar, A., Terry, B., Li, F., Zhang, J., & Shao, S. (2024) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars61786.2024.10778676 · Cited by 10
Microservice logs analysis employing AI: A systematic literature review
Uddin, A., Weerasinghe, S., Gajewski, D., Akbarsharifi, M., Akbarsharifi, R., Stoner, C., Cerny, T., & He, S. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112786 · Cited by 11
Sprelog: Log-Based Anomaly Detection with Self-matching Networks and Pre-trained Models
Yang, H., Zhao, X., Sun, D., Wang, Y., & Huang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91431-8_50 · Cited by 7
Log-based Anomaly Detection of Enterprise Software: An Empirical Study
Wijesinghe, N., & Hemmati, H. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00012 · Cited by 8
“A different approach was taken in another experiment, where anomaly detection was modeled as a dimensionality reduction problem [18].”
MethodCorbelle, C., Carneiro, V., & Cacheda, F. (2024) · Applied Sciences · doi:10.3390/app14135388 · Cited by 9
“By applying hierarchical classification to system logs, thematic trees can be used to analyze and categorize log messages more efficiently [13].”
MethodLightweight Multi-Task Learning Method for System Log Anomaly Detection
Pham, T.-A., & Lee, J. (2024) · IEEE Access · doi:10.1109/access.2024.3425369 · Cited by 8
“Machine learning-based approaches such as SVM [6], IM [7], and PCA [8] primarily count message types for every window session and group them into message count vectors as invariant features for training and detection.”
MethodLuo, X., Wang, J. J., Yin, L., Zhao, K., Qian, K., Zhang, D., & Chen, K. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3491894 · Cited by 7
“We employed the HDFS dataset [43] and a large real-world dataset to evaluate the proposed approach BiCAM.”
Anomaly detection in virtual machine logs against irrelevant attribute interference
Zhang, H., Zhou, Y., Xu, H., Shi, J., Lin, X., & Gao, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315897 · Cited by 7
Domain Tailored Large Language Models for Log Mask Prediction in Cellular Network Diagnostics
Taheri, S., Ihalage, A. A., Mishra, P., Coaker, S., Faris, M., & Al-Raweshidy, H. S. (2025) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2025.3541384 · Cited by 7
Dong, W., Luo, L., Chen, C., Bu, J., Liu, X., & Liu, Y. (2016) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2016.2542815 · Cited by 7
“Previous network-level diagnosis methods only detect network problems at the node level or link level [15], [18], D2 is able to point programmers closer to the most likely causes by a novel approach combining statistical tests and program call graph analysis.”
BackgroundDynamic Logging with Dylog in Networked Embedded Systems
Dong, W., Luo, L., & Huang, C. H. (2015) · ACM Transactions on Embedded Computing Systems · doi:10.1145/2807698 · Cited by 7
“The availability of the event logs is a prerequisite for advanced diagnosis and analysis (e.g., frequent pattern mining [Khan et al. 2008], principal component analysis [Xu et al. 2009; Dong et al. 2013]) at the backend.”
DriftInsight: Detecting Anomalous Behaviors in Large-Scale Cloud Platform
Meng, F. J., Xiao, Z., Chen, P., & Xu, J. M. (2017) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2017.37 · Cited by 7
“[14] extracts feature from logs, and then uses PCA to discover the invariances between the features.”
MethodIntelligent Log Analysis Using Machine and Deep Learning
Yen, S., & Moh, M. (2019) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-5225-8100-0.ch007 · Cited by 7
Transfer Log-based Anomaly Detection with Pseudo Labels
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm50824.2020.9269069 · Cited by 7
“We evaluated our proposed method on three log datasets including HDFS [9], the BGL dataset [10], and the OpenStack dataset [11], where taking HDFS dataset and BGL dataset as labeled source domain and the OpenStack dataset as unlabeled target domain.”
MethodLogFlow: Simplified Log Analysis for Large Scale Systems
Platini, M., Ropars, T., Pelletier, B., & Palma, N. D. (2021) · International Conference on Distributed Computing and Networking (ICDCN) · doi:10.1145/3427796.3427808 · Cited by 9
“This is in general realistic, as logs are generated based on message templates included in the code of applications, libraries, OS services, etc. [24].”
BackgroundInfluentialYou, C., Wang, Q., & Sun, C. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82199-9_52 · Cited by 8
Feature Selection for Fault Detection and Prediction based on Event Log Analysis
Li, Z., & van Leeuwen, M. (2022) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3575637.3575652 · Cited by 7
“The traditional benchmark datasets for log anomaly detection, including HDFS [16], BGL [12], Spirit [12] and Thunderbird [12], are mainly produced by complex systems such as cloud service providers or supercomputers.”
BackgroundTowards Detecting Anomalies in Log-Event Sequences with Deep Learning: Open Research Challenges
Himler, P., Landauer, M., Skopik, F., & Wurzenberger, M. (2023) · European Interdisciplinary Cybersecurity Conference (EICC) · doi:10.1145/3590777.3590789 · Cited by 7
“Our approach has been tested with the HDFS data set [26] and also an Audit data set [15].”
MethodInfluentialDetect anomalies in cloud platforms by using network data: a review
Jayaweera, M. P. G. K., Kithulwatta, W. M. C. J. T., & Rathnayaka, R. M. K. T. (2023) · Cluster Computing · doi:10.1007/s10586-023-04055-1 · Cited by 8
DeepUserLog: Deep Anomaly Detection on User Log Using Semantic Analysis and Key-Value Data
Wang, W., Lu, S., Luo, J., & Wu, C. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00046 · Cited by 7
“…3) Existing methods : DeepUserLog is a learning-based method that we compared to various other methods, including Principal Component Analysis(PCA) [16], Invariants Mining (IM) [17], SVM [18], Decision Tree [19], LogClustering [20], DeepLog [3], LogAnomaly [9], LogRobust [10], and NeuralLog [11].”
MethodUnveiling anomalies: transformative insights from transformer-based autoencoder models
Dehkordi, S. B., Nasri, S., & Dami, S. (2024) · International Journal of Computers and Applications · doi:10.1080/1206212x.2024.2441147 · Cited by 9
“Numerouslong-enduranceunmannedaerialvehicles(UAVs) havebeencreatedandusedinthecontextofinspectionactivities [20,26,41,42].”
Interactive learning of alert signatures in High Performance Cluster system logs
Makanju, A., Zincir-Heywood, N., & Milios, E. (2012) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2012.6211882 · Cited by 7
“Visualization techniques, if included, are not used interactively and are used only for the visualization of the results of analysis [4], [5].”
BackgroundMethodInfluentialMakanju, A., Zincir-Heywood, N., & Milios, E. (2011) · ACM Symposium on Applied Computing · doi:10.1145/1982185.1982298 · Cited by 7
“Previous approaches to the task of message type extraction include SLCT and Loghound [15], extraction from source code [16], sequential mining from an online stream [2] and IPLoM (Iterative Partitioning Log Mining) [5].”
BackgroundMethodInfluentialEvaluation of in‐memory storage engine for machine learning analysis of security events
Sapegin, A., Gawron, M., Jaeger, D., Cheng, F., & Meinel, C. (2016) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.3800 · Cited by 7
“Xu et al. use message types, message count vectors and state ratio (e.g. COM-MITTING/ABORTING) as features [35]; N. Al-Rousan counts different types of Border Gateway Protocol (BGP) updates for BGP root cause analysis [36].”
MethodLiu, X., Tong, Y., Xu, A., & Akkiraju, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-76352-7_18 · Cited by 8
Unsupervised Anomaly Detection Based on System Logs
Chen, H. (2021) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2021-126 · Cited by 7
“EXPERIMENT A. Dataset 1) Stable Dataset: We conduct our experiment on stable log dataset HDFS [2] and BGL [16].”
MethodInfluentialLeveraging Log Instructions in Log-based Anomaly Detection
Bogatinovski, J., Madjarov, G., Nedelkoski, S., Cardoso, J., & Kao, O. (2022) · IEEE International Conference on Services Computing · doi:10.1109/scc55611.2022.00053 · Cited by 12
“apply PCA [10] to learn the normal state of the event counts by projecting them as points in a vector space.”
BackgroundMethodInfluentialRoundtable: The Future of Software Engineering for Internet Computing
Lü, J., Rosenblum, D. S., Bultan, T., Valerie Issarny, Z., Dustdar, S., Storey, M., & Zhang, D. (2015) · IEEE Software · doi:10.1109/ms.2015.15 · Cited by 7
Prediction of failure occurrence time based on system log message pattern learning
Sonoda, M., Watanabe, Y., & Matsumoto, Y. (2012) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2012.6211960 · Cited by 7
A Markov Random Field Based Approach for Analyzing Supercomputer System Logs
Hacker, T. J., Pais, R., & Rong, C. (2017) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2017.2678473 · Cited by 7
GenLog: Accurate Log Template Discovery for Stripped X86 Binaries
Zhang, M., Zhao, Y., & He, Z. (2017) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2017.137 · Cited by 7
“Therefore automatic analysis of logs has become an important research problem that has drawn considerable attentions [2], [16]–[20], [25], [26], [34], [37], [38].”
BackgroundEfficient tamper-evident logging of distributed systems via concurrent authenticated tree
Ning, F., Wen, Y., Shi, G., & Meng, D. (2017) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/pccc.2017.8280476 · Cited by 7
“For example, if a Byzantine machine provided incomplete log data to the analysis system, the fault diagnosis program would incorrectly attribute the cause of an operational failure [2].”
BackgroundA Standardized ICS Network Data Processing Flow With Generative Model in Anomaly Detection
Yang, T., Hu, Y., Li, Y., Hu, W., & Pan, Q. (2019) · IEEE Access · doi:10.1109/access.2019.2963144 · Cited by 8
“In view of the reliability and stability of anomaly detection systems (ADS) in traditional IT network [6], [7], many researches on ADS for ICS network has gradually emerged.”
BackgroundCreating Character-based Templates for Log Data to Enable Security Event Classification
Wurzenberger, M., Höld, G., Landauer, M., Skopik, F., & Kästner, W. (2020) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3320269.3384722 · Cited by 7
“The third data set, we refer to as DSC, includes logs from a Hadoop File System running on a 203node cluster on Amazon’s EC2 platform [13].”
MethodWhat Distributed Systems Say: A Study of Seven Spark Application Logs
Gholamian, S., & Ward, P. A. S. (2021) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds53918.2021.00030 · Cited by 12
ADP: automated diagnosis of performance pathologies using hardware events
Yoo, W., Larson, K., Baugh, L., Kim, S. K., & Campbell, R. H. (2012) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/2318857.2254791 · Cited by 7
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.