Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 701–800 of 1,004
Finding Needles in the Haystack: Harnessing Syslogs for Data Center ManagementPreprint
Liang, C., Benson, T., Kanuparthy, P., & He, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06150 · Cited by 7
LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs
Hadadi, F., Xu, Q., Bianculli, D., & Briand, L. (2024) · arXiv · doi:10.48550/arxiv.2406.07467 · Cited by 7
“The majority of anomaly detection methods [20, 24, 50, 56, 61, 64, 90, 92] have been proposed for and evaluated on stable log datasets.”
System log anomaly detection based on contrastive learning and retrieval augmented
Li, W., Wu, Y., Huang, W., Zhou, F., Ou, W., Wang, H., & Deng, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-22208-7 · Cited by 6
Talasso, G. U., Souza, A. M. D., Guidoni, D. L., Cerqueira, E., & Villas, L. A. (2025) · doi:10.5753/sbrc.2025.5859 · Cited by 6
Dataflow Feature Analysis for Industrial Networks Communication Security
Zhang, D., Hu, Y., Cao, G., Liu, Y., Shi, Y., Minghao, H., & Pan, Q. (2020) · Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University · doi:10.1051/jnwpu/20203810199 · Cited by 6
Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
Yang, Y., Wang, B., & Zhao, C. (2023) · Applied Sciences · doi:10.3390/app13063691 · Cited by 6
Rong, G., Gu, S., Shen, H., Zhang, H., & Kuang, H. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00080 · Cited by 11
DSGN: Log-based anomaly diagnosis with dynamic semantic gate networks
Yang, H., Sun, D., Wang, Y., & Huang, W. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121174 · Cited by 6
Substructure-Aware Log Anomaly Detection
Tang, Y., Zhang, Z., Zhao, K., Fang, L., Li, Z., & Chen, W. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3705829.3705840 · Cited by 11
A Multi-Source Log Hidden Anomaly Detection Method Integrating Trans-Encoder and LSTM
Luo, C., Jiang, H., Zhou, P., Wang, X., Shao, Z., & Sun, W. (2025) · IEEE Access · doi:10.1109/access.2025.3556959 · Cited by 6
“This was the first large dataset used in an online PCA-based approach [17] and has been used in several subsequent works, including online PCA approaches [18] and IM-based approaches [19].”
Cheng, K., Tang, W., Tan, L., Yang, J., & Chen, J. (2025) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2025.3605764 · Cited by 8
“We use two datasets from Loghub [43] as experimental datasets, HDFS [44], BGL [45].”
Detection Anomalies Bid Data in Unstructured Syslogs
Sheluhin, O. I., & Ryabinin, V. (2019) · Voprosy kiberbezopasnosti · doi:10.21681/2311-3456-2019-2-36-41 · Cited by 6
Building Golden Signal Based Signatures for Log Anomaly Detection
Nagar, S., Samanta, S., Mohapatra, P., & Kar, D. (2022) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud55607.2022.00040 · Cited by 8
“HDFS Dataset: We also used logs collected from a real world dataset, the Hadoop Distributed File System (HDFS) [18].”
MethodServerRCA: Root Cause Analysis for Server Failure using Operating System Logs
Shi, J., Jiang, S., Xu, B., & Xiao, Y. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00083 · Cited by 8
Bouricha, H., Arwa, B., Hsairi, L., & Ahmadi, H. A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122122 · Cited by 6
LogTransformer: Transforming IT System Logs Into Events Using Tree-Based Approach
Fu, Y., & Xu, J. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3391290 · Cited by 7
“The evolution of the parsing approach can be summarized as a shift from rule-based approaches [22], [23] and source code-based approaches [24], [25] to data-driven approaches [26], [27].”
MethodPosParser: A Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging
Jiang, J., Fu, Y., & Xu, J. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3453756 · Cited by 6
“Xu et al. [33] first convert the source code into an abstract syntax tree (AST), then traverses the AST to detect log output statements and construct the toString table and type hierarchy table, and finally reconstruct the log event template.”
LogCTBL: a hybrid deep learning model for log-based anomaly detection
Huang, H., Luo, W., Wang, Y., Zhou, Y., & Huang, W. T. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-06926-3 · Cited by 6
“PCA [31].”
Hu, J., Long, L., Sui, H., Gu, Z., & Zheng, G. (2025) · Applied Sciences · doi:10.3390/app15041740 · Cited by 6
MidLog: An automated log anomaly detection method based on multi-head GRU
Yuan, W., Ying, S., Duan, X., Cheng, H., Zhao, Y., & Shang, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112431 · Cited by 9
A Density-Based Anomaly Detection Method for MapReduce
Wang, K., Wang, Y., & Yin, B. (2012) · IEEE International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca.2012.15 · Cited by 6
Dexter: faster troubleshooting of misconfiguration cases using system logs
Talwadker, R. (2017) · ACM International Systems and Storage Conference (SYSTOR) · doi:10.1145/3078468.3078484 · Cited by 6
“[49] applied machine learning techniques to learn common patterns from a large amount of console logs.”
BackgroundA hybrid approach for log signature generation
Pokharel, P., Pokhrel, R., & Joshi, B. (2019) · Applied Computing and Informatics · doi:10.1016/j.aci.2019.05.002 · Cited by 6
A Study on Log Anomaly Detection using Deep Learning Techniques
Pithode, K., & Patheja, P. S. (2022) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic53929.2022.9793238 · Cited by 9
“on human log review and is no longer adequate due to the following factors [3].”
BackgroundNatural Language Processing-based Model for Log Anomaly Detection
Li, Z., Zhang, J., Zhang, X., Lin, F., Wang, C., & Cai, X. (2022) · IEEE International Conference on Software Engineering and Artificial Intelligence (SEAI) · doi:10.1109/seai55746.2022.9832400 · Cited by 9
Log Anomaly Detection Based on Hierarchical Graph Neural Network and Label Contrastive Coding
Fang, Y., Zhao, Z., Xu, Y., & Liu, Z. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.033124 · Cited by 6
On Data Processing through the Lenses of S3 Object Lambda
Sarroca, P. G., & Sánchez‐Artigas, M. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228890 · Cited by 7
“The other is IO-bound, and invokes a grep function to return all the lines that contain the word error in a HDFS log [23].”
BackgroundHEART: Heterogeneous Log Anomaly Detection Using Robust Transformers
Mvula, P. K., Branco, P., Jourdan, G.-V., & Viktor, H. L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45275-8_45 · Cited by 6
Event log anomaly detection method based on auto-encoder and control flow
Kan, D., & Fang, X. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01199-3 · Cited by 6
“PCA: We use PCA to detect anomalies in the event log based on the approach used in the literature [36].”
Study in Usefulness of Middleware-Only Provenance
Zhou, Q. Z., Ghoshal, D., & Plale, B. (2014) · IEEE International Conference on eScience · doi:10.1109/escience.2014.49 · Cited by 6
“[20] proposes a mechanism to mine logs combined with the source code that generated the logs to detect problems in large scale systems.”
BackgroundCAPRI: a tool for mining complex line patterns in large log data
Zulkernine, F. H., Martin, P., Powley, W., Soltani, S., Mankovskii, S., & Addleman, M. (2013) · International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications (BigMine) · doi:10.1145/2501221.2501228 · Cited by 6
Large Scale Data Analytics of User Behavior for Improving Content Delivery
Balachandran, A. (2014) · Cited by 6
VEGAS: Visualizing, exploring and grouping alerts
Crémilleux, D., Bidan, C., Majorczyk, F., & Prigent, N. (2016) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2016.7502968 · Cited by 6
“The PCA algorithm is also used in [14]: the authors apply this technique to detect runtime problem and classify errors.”
MethodAn End-To-End Log Management Framework for Distributed Systems
He, P. (2017) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2017.41 · Cited by 7
“tion [3], program verification [4], problem identification [5], etc.”
BackgroundSoftware Troubleshooting Using Machine Learning
Kalibhat, N., Varshini, S. R., Kollengode, C., Sitaram, D., & Kalambur, S. (2017) · IEEE International Conference on High Performance Computing Workshops (HiPCW) · doi:10.1109/hipcw.2017.00010 · Cited by 6
Event Extraction from Streaming System Logs
Guo, S., Liu, Z., Chen, W., & Li, T. (2018) · Lecture notes in electrical engineering · doi:10.1007/978-981-13-1056-0_47 · Cited by 9
Enhancing HPC System Log Analysis by Identifying Message Origin in Source Code
Fulp, M. H., Fulp, D., Baseman, E., Blanchard, S. P., Greenberg, H., Jones, W. M., & DeBardeleben, N. (2018) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2018.00-23 · Cited by 7
User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications
Itkin, I., Gromova, A., Sitnikov, A., Legchikov, D., Tsymbalov, E., Yavorskiy, R., Novikov, A., & Rudakov, K. (2019) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest.2019.000-9 · Cited by 7
“For anomalies search, there are three categories of methods generally used: PCA based approaches [24], invariant mining based methods [25], [26] and workflow-based methods [27].”
MethodSecurity of HPC Systems: From a Log-analyzing Perspective
Luo, Z., Qu, Z., Nguyen, T. T., Hui, Z., & Lu, Z. (2019) · ICST Transactions on Security and Safety · doi:10.4108/eai.19-8-2019.163134 · Cited by 9
“• Xu et al. [46] o ff ered a method leveraging the source code to parse log files, in which we need to first get all possible log message template strings from the source code and then match it to log files to parse them into structured files.”
MethodA Confidence-Guided Evaluation for Log Parsers Inner Quality
Xie, X., Wang, Z., Xiao, X., Lu, Y., Huang, S., & Li, T. (2020) · Mobile Networks and Applications · doi:10.1007/s11036-019-01501-6 · Cited by 6
“Through log analysis, engineers can discover and detect system anomalies [5–7], trace system or user behaviors [8] and prevent malicious attacks.”
BackgroundFastLogSim: A Quick Log Pattern Parser Scheme Based on Text Similarity
Liu, W., Liu, X., Di, X., & Cai, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55130-8_19 · Cited by 6
Wang, J., Liu, C., Xu, J., Wang, J., Hao, S., Yi, W., & Jing, Z. (2022) · Security and Communication Networks · doi:10.1155/2022/1443978 · Cited by 9
“[46] proposed an unsupervised method for detecting anomalous sequences of events using PCA, but PCA is data-sensitive so the detection accuracy of PCA will vary on different data sets.”
MethodGurumdimma, N. Y., Jhumka, A., Liakata, M., Chuah, E., & Browne, J. C. (2015) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom.2015.613 · Cited by 6
Utilizing Persistence for Post Facto Suppression of Invalid Anomalies Using System Logs
Guhathakurta, D., Aggarwal, P., Nagar, S., Arora, R., & Zhou, B. (2022) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3510455.3512774 · Cited by 7
“More details on the HDFS dataset can be found in [31].”
BackgroundAirphant: Cloud-oriented Document Indexing
Chockchowwat, S., Sood, C., & Park, Y. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00107 · Cited by 7
Utility and accuracy of smell-driven performance analysis for end-user programmers
Chambers, C., & Scaffidi, C. P. (2014) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2014.10.017 · Cited by 6
Malikov, A., Авраменко, В., & Saenko, I. B. (2019) · Информационно-управляющие системы · doi:10.31799/1684-8853-2019-6-32-42 · Cited by 5
“Альтернативный метод обработки журналов событий предложен в работах [18, 19].”
PHOENIX : Misconfiguration Detection for AWS Serverless Computing
Wen, J., & Ping, H. (2025) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2025.3577211 · Cited by 5
“Weapplyexistingassociation ruleminingalgorithms[18],[35],specificallythestate-of-the-art FP-Growth[36],duetoitsefficiency,scalability,andmaturity inminingfrequentitemsetsfromstructuredconfigurationdata.”
Latent Fault Detection With Unbalanced Workloads
Gabel, M., Sato, K., Keren, D., Matsuoka, S., & Schuster, A. (2015) · Cited by 5
CloudSeer: Workflow Monitoring of Cloud Infrastructures via Interleaved Logs
Yu, X., Joshi, P., Xu, J., Jin, G., Zhang, H., & Jiang, G. (2016) · ACM SIGOPS Operating Systems Review · doi:10.1145/2954680.2872407 · Cited by 5
Boosted CSIRT with AI powered open source framework
Maezo, R. G., & Rey, A. E. (2023) · doi:10.23919/jnic58574.2023.10205787 · Cited by 5
“According to a study by Oliner, Adam and Ganapathi et al. [1], log analysis tools has been widely employed to improve the reliability of software systems in many aspects, such as anomaly detection [2], [3], failure diagnosis, [4], [5] program verification, [6], [7] performance prediction.”
MethodBhanage, D. A., Pawar, A., Kotecha, K. V., & Abraham, A. (2023) · IEEE Access · doi:10.1109/access.2023.3319438 · Cited by 6
“indexed-based methods [19], [20], [21] or semantic-based methods [22], [23], [24] to extract the features of logs.”
BackgroundMethodInfluentialLogGC: Novel Approach for Graph-based Log Anomaly Detection
Andonov, S., & Madjarov, G. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00156 · Cited by 6
“The HDFS dataset is created in [25] by collecting the AWS logs generated by the EC2 (Elastic Compute Cloud) instances on which Hadoop map-reduce jobs were run.”
MethodDualAttlog: Context aware dual attention networks for log-based anomaly detection
Yang, H., Sun, D., & Huang, W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106680 · Cited by 6
Log-Based Anomaly Detection with Transformers Pre-Trained on Large-Scale Unlabeled Data
Yan, S., Shi, L., Ren, J., Wang, W., Liu, Y., Sun, L., Wang, X., & Zhang, W. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10623067 · Cited by 5
“For ML-based methods, we evaluate PCA [14], Isolation Forest [8], and LogCluster [7] implemented in Loglizer [2].”
MethodCross-Domain Graph Level Anomaly Detection
Li, Z., Liang, S., Shi, J., & van Leeuwen, M. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3462442 · Cited by 5
“Hence, following [2], we construct four graph datasets: HDFS (HD) [44], BGL (BG), SPIRIT (SP), and THUNDER-BIRD (TH) [45], where each dataset contains 5000 graphs with 5% anomalous graphs.”
MethodLogOW: A semi-supervised log anomaly detection model in open-world setting
Ye, J., Liu, C., Gu, Z., Zhang, Z., Meng, X., Zhang, W., & Zhang, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112305 · Cited by 7
Han, D., Wang, Z., Feng, R., Jin, M., Chen, W., Wang, K., Wang, S., Yang, J., Shi, X., Yin, X., et al. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670375 · Cited by 11
Understanding the Linux Kernel, Visually
Liu, H., Jiang, Y., & Xu, C. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696095 · Cited by 5
“Various automated debugging solutions have been proposed, including delta debugging [66, 85, 91], statistical debugging [30, 53, 61, 70, 89], causal tracing [42, 52, 63, 86], and reverse debugging [37, 45, 84, 90].”
ELFA-Log: Cross-System Log Anomaly Detection via Enhanced Pseudo-Labeling and Feature Alignment
Zhao, X., Guo, K., Huang, M., Qiu, S., & Lu, L. (2025) · Computers · doi:10.3390/computers14070272 · Cited by 6
Scrub: online troubleshooting for large mission-critical applications
Satish, A., Shiou, T., Zhang, C., Elmeleegy, K., & Zwaenepoel, W. (2018) · European Conference on Computer Systems · doi:10.1145/3190508.3190513 · Cited by 5
“Our work was in large part motivated by the fact that logging [31, 37, 38, 43, 44, 46] – in practice is still the most common troubleshooting technique for distributed systems – is inadequate for our environment.”
BackgroundA Confidence-Guided Anomaly Detection Approach Jointly Using Multiple Machine Learning Algorithms
Xie, X., Jin, Z., Han, Q., Huang, S., & Li, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37352-8_8 · Cited by 5
Frikha, M. S., Gammar, S. M., & Lahmadi, A. (2021) · doi:10.23919/pemwn53042.2021.9664667 · Cited by 5
“However, traditional anomaly detection techniques [11]–[13] cannot be suitable for sensor-based IoT networks due to their resource and communication constraints, large-scale and dynamic topology, etc.”
Unstructured Log Analysis for System Anomaly Detection—A Study
Behera, A., Panigrahi, C. R., & Pati, B. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-5685-9_48 · Cited by 6
LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing
Li, D., Zhang, J., Zhang, X., Lin, F., Wang, C., & Chang, L. (2022) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct56141.2022.10072770 · Cited by 5
AdaptParse: Adaptive Contextual Aware Attention Network for Log Parsing via Word Classification
Yang, H., Sun, D., Wang, Y., Zhao, N., Zhang, S., & Huang, W. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191155 · Cited by 5
“Due to the dynamic nature of unstructured logs [15], [16], automated log analysis often has difficulties in analyzing and processing unstructured logs.”
BackgroundMethodSemi-supervised Power Microservices Log Anomaly Detection Based on BiLSTM and BERT with Attention
Dai, Z., Yong, L., Chen, M., Chen, L., Fang, W., & Ziang, L. (2023) · International Conference on Advanced Electronics, Electrical and Green Energy (AEEGE) · doi:10.1109/aeege58828.2023.00023 · Cited by 5
“Taking the Hadoop distributed file system as an example, this system can generate more than 24 million logs within 48 hours [1].”
MethodInfluentialGraph-Based Log Anomaly Detection via Adversarial Training
He, Z., Tang, Y., Zhao, K., Liu, J., & Chen, W. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8664-4_4 · Cited by 8
LGLog: Semi-supervised Graph Representation Learning for Anomaly Detection based on System Logs
Liu, J., Tang, Y., Liu, J., Zhao, K., & Chen, W. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00014 · Cited by 5
“To address the above problems, many scholars started to come up with approaches based on traditional machine learning, such as PCA [7], invariant mining [8], clustering [9], but these methods are under-performing since they fail to fully mine the rich semantic information contained in logs and the…”
BackgroundMethodIhalage, A. A., Taheri, S., Faris, M., & Al-Raweshidy, H. S. (2025) · IEEE Access · doi:10.1109/access.2025.3587029 · Cited by 5
“Xu et al. [32] represented log sequences as event count vectors and used PCA to obtain a lower dimensional representation of logs.”
Provenance for System Troubleshooting.
Chiarini, M. (2011) · USENIX Large Installation Systems Administration Conference · Cited by 6
Schema Independent Reduction of Streaming Log Data
Kalamatianos, T., & Kontogiannis, K. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07881-6_27 · Cited by 5
Ansari, S., Mohanlal, R., Poncela, J., Ansari, A., & Mohanlal, K. (2015) · Advances in data mining and database management book series · doi:10.4018/978-1-4666-8505-5.ch001 · Cited by 5
Communication Network Anomaly Detection Based on Log File Analysis
Cheng, X., & Wang, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11740-9_23 · Cited by 5
“Due to limitations of space, we refer readers unfamiliar with these techniques to the work [10] for details.”
MethodImproving Logging Prediction on Imbalanced Datasets: A Case Study on Open Source Java Projects
Lal, S., Sardana, N., & Sureka, A. (2016) · International Journal of Open Source Software and Processes · doi:10.4018/ijossp.2016040103 · Cited by 7
“Xu et al. (2009) propose a method for mining log messages to detect system runtime problems.”
MethodAn Analysis of Burstiness and Causality of System Logs
Otomo, K., Kobayashi, S., Fukuda, K., & Esaki, H. (2017) · Asian Internet Engineering Conference · doi:10.1145/3154970.3154973 · Cited by 8
“To detect system anomaly with system logs, there are a lot of existing methods [14, 19].”
MethodLiu, Z., Guan, X., Li, S., Qin, T., & He, C. (2018) · IEEE Access · doi:10.1109/access.2018.2882812 · Cited by 5
“[14], [15] extract the correlations between the logs and apply principal component analysis (PCA) to detect anomalies, then construct decision trees to classify the detected anomalies.”
MethodDetecting Anomalies in Cluster System Using Hybrid Deep Learning Model
Xiao, C., Huang, J., & Wu, W. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-2767-8_35 · Cited by 6
Predicting Hadoop misconfigurations using machine learning
Robert, A., Gupta, A., Shenoy, V. S., Sitaram, D., & Kalambur, S. (2020) · Software Practice and Experience · doi:10.1002/spe.2790 · Cited by 5
“On a similar note, Xu et al 37 used machine learning on features extracted from console logs.”
MethodUnsupervised Noise Detection in Unstructured data for Automatic Parsing
Jain, S., de Buitléir, A., & Fallon, E. (2020) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm50824.2020.9269096 · Cited by 5
“…analysis techniques are not accurate and further that even slight error in pre-processing can cause a huge performance degradation on applications such as anomaly detection [11] [12] [1], cognitive management [2], autonomous problem validation [13] [3], incident management or root cause analysis.”
MethodLog-Based Malicious Activity Detection Using Machine and Deep Learning
Tarnowska, K. A., & Patel, A. (2020) · Springer eBooks · doi:10.1007/978-3-030-62582-5_23 · Cited by 6
Trace reconstruction in system logs for processing with process mining
Jürgensen, J. P. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.01.173 · Cited by 5
Anomaly Detection Mechanism Based on Hierarchical Weights through Large-Scale Log Data
Xi, L., Yang, X. S., Luo, S., Shang, Y., & Tang, Q. (2021) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai50917.2021.9447458 · Cited by 5
“Existing typical approaches of automated log-based anomaly detecting are: (1) Principal Component Analysis(PCA) based, in [1], they formulated the log-based anomaly detection problem as an unsupervised learning problem; (2) invariant mining based methods, for example, in [2], they captured…”
MethodInfluentialReview on Log-Based Anomaly Detection Techniques
Raut, P., Mishra, A., Rao, S., Kawoor, S., Shelke, S., Deore, M. P., & Kumar, V. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-7657-4_72 · Cited by 5
Diagnosing Kernel Concurrency Failures with AITIA
Jeong, D. R., Jung, M., Lee, Y., Lee, B., Shin, I., & Kwon, Y. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3567486 · Cited by 5
“A huge number of prior works fall into the same category called cooperative bug localization [7, 8, 13, 34, 35, 37, 38, 43, 56, 108].”
BackgroundOnline log parsing using evolving research tree
Vervaet, A., Callau-Zori, M., Chabchoub, Y., & Chiky, R. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01953-z · Cited by 5
“In recent years, several solutions were proposed based on different techniques: Principal component analysis [29], invariant mining [30], LSTM networks [31], and BERT transformer architecture [32].”
KAD: a knowledge formalization-based anomaly detection approach for distributed systems
Wei, X., Sun, C., & Zhang, X. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09670-8 · Cited by 5
“Xu et al. (2009) constructed normal space and abnormal space of log event count matrix using principal component analysis to detect anomalies.”
Challenges to error diagnosis in hadoop ecosystems
Li, J., He, S., Zhu, L., Xu, X., Fu, M., Bass, L., liu, A., & Tran, A. B. (2013) · Cited by 5
Optimizing apache nutch for domain specific crawling at large scale
Lopez, L. A., Duerr, R. E., & Khalsa, S. S. (2015) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2015.7363976 · Cited by 5
Log Based Analysis of Software Application Operation
Obrębski, D., & Sosnowski, J. S. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-19501-4_37 · Cited by 5
Anomaly Detection with HMM Gauge Likelihood Analysis
Lorbeer, B., Deutsch, T., Ruppel, P., & Küpper, A. (2019) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice.2019.00008 · Cited by 5
“One way of implementing this is by using PCA on event counts (Xu et al., 2010a).”
BackgroundMethodAbnormal Subspace Sparse PCA for Anomaly Detection and InterpretationPreprint
Bin, X., Zhao, Y., & Shen, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.04644 · Cited by 5
“Wei Xu et al. also tried to interpret the results returned by a PCA anomaly detection model with decision trees trained by the data labeled by the PCA model [22], which as shown in our experimental results, can be misleading and fail to reveal the true reason behind the PCA model.”
InfluentialSTEP: Spatial-Temporal Network Security Event PredictionPreprint
Cheng, Q., Shen, Y., Kong, D., & Wu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14932 · Cited by 5
“In contrast, there are fewer ’0 event’ in the HDFS data set, and the accuracy of the prediction is more suitable for the actual scenario.”
MethodInfluentialBorrowing from Similar Code: A Deep Learning NLP-Based Approach for Log Statement AutomationPreprint
Gholamian, S., & Ward, P. A. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01259 · Cited by 5
“and system administrators will review at a later time for various purposes, such as anomaly and problem detection [25, 75], log message clustering [53, 70], system profile building, code quality assessment [66], and code coverage analysis [22].”
BackgroundSystem Log Anomaly Detection With Noise-Contrastive Learning and Pattern Feature
Luo, P., Deng, D., Xie, M., Yang, G., Chu, J., Soong, B., & Yuen, C. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3579809 · Cited by 4
“Out of a total of 111,979,954 log entries compiled, approximately 2.9% were classified as anomaly samples, encompassing incidents like “write exceptions” [41].”
InfluentialSSDALog: Semi-Supervised Domain Adaptation for Incremental Log-Based Anomaly Detection
Tian, J., Li, M., Chen, L., Wang, Z., Nie, X., & Qin, J. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3583483 · Cited by 5
“Traditional machine learning methodologies [6], [7] often struggle to accurately capture the complex dependencies among events during log data analysis, thus limiting their e ffi cacy for log anomaly detection.”
SemanticLog: Towards Effective and Efficient Large-Scale Semantic Log Parsing
Zhang, C., Xu, W., Liu, J., Zhang, L., Liu, G., Guan, J., Zhou, Q., & Zhou, S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3625121 · Cited by 7
“On the other hand, with the popularity of artificial intelligence for IT operations (AIOPs), traditional log parsers can no longer meet the needs of many downstream tasks such as anomaly detection [5], [6], [7], [8], root cause analysis [27], [28], [29], and log summarization [30], [31], [32].”
Self-citationA survey of anomaly detection in HPC systems using machine learning
Lu, Y., Zhang, W., Yang, L., Dai, Y., Wu, H., Wu, Z., Tian, H., Li, Y., Chen, J. B., Liu, C., et al. (2026) · CCF Transactions on High Performance Computing · doi:10.1007/s42514-025-00266-7 · Cited by 4
“Their proposed method was tested on three datasets: HDFS (Xu et al. 2009 ), BGL (Oliner and Stearley 2007 ), and OpenStack (Du et al. 2017 ).”
Masri, Y., Ma, E., Wang, Z., Rogers, J., & Yang, C. P. (2026) · Computers · doi:10.3390/computers15070430 · Cited by 4
Review of Anomaly Detection Based on Log Analysis
Wu, X. T. (2020) · International Journal of Advanced Network Monitoring and Controls · doi:10.21307/ijanmc-2020-036 · Cited by 5
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.