Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 801–900 of 1,004
Machine Learning Toolkit for System Log File Reduction and Detection of Malicious Behavior
Ritchey, R. P., & Perry, R. (2021) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps51825.2021.9484572 · Cited by 4
Anomaly Detection in the Key-Management Interoperability Protocol Using Metadata
Baee, M. A. R., Simpson, L. R., & Armstrong, W. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3386715 · Cited by 4
“These are three unsupervised methods (Principal Component Analysis (PCA) [19], Invariant Mining [20], and Log Clustering [21]) and three supervised methods (Decision Tree [22], Support Vector Machines (SVM) [23], and Logistic Regression [24]).”
MethodAnomaly Detection in Key-Management Activities Using Metadata: A Case Study and Framework
Baee, M. A. R., Simpson, L. R., & Armstrong, W. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3407547 · Cited by 4
“These are three unsupervised methods (Principal Component Analysis (PCA) [18], Invariant Mining [19], and Log Clustering [20]) and three supervised methods (Decision Tree [21], Support Vector Machines (SVM) [22], and Logistic Regression [23]).”
MethodUnsupervised Anomaly Detection on Distributed Log Tracing through Deep Learning
Khudyakov, D. A., & Эркиновна, Я. Г. (2024) · International Siberian workshops and tutorials on electron devices and materials · doi:10.1109/edm61683.2024.10615125 · Cited by 5
“The first solutions in the field of log analysis were based on the use of statistical methods of machine learning [7], [8].”
MethodEnhancing Log Anomaly Detection with Semantic Embedding and Integrated Neural Network Innovations
Xu, Z., Wang, Z., Xu, J. F., Shi, H., & Zhao, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.051620 · Cited by 8
Performance issue monitoring, identification and diagnosis of SaaS software: a survey
Wang, R., Tian, X., & Ying, S. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-023-2701-0 · Cited by 4
System Logs Anomaly Detection. Are we on the right path?
Albert, R.-G. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2440692 · Cited by 5
“They are split into two categories: distributed systems with HDFS (Xu et al. 2009), and supercomputers with BGL and Spirit, datasets collected by Sandia National Labs (SNL) (Oliner and Stearley 2007).”
Distributed System Log Anomaly Detection Method Based on LSTM Networks and Process State Inspection
Wang, J., Hu, J., & Li, P.-D. (2025) · Quality and Reliability Engineering International · doi:10.1002/qre.3793 · Cited by 4
The Performance Mining Method: Extracting Performance Knowledge from Software Operation Data
Pachidi, S., & Spruit, M. R. (2015) · International Journal of Business Intelligence Research · doi:10.4018/ijbir.2015010102 · Cited by 4
“Xu et al. (2009) suggest mining techniques for analyzing console logs in order to detect problems in large-scale systems such as online games.”
BackgroundA hierarchical reliability-driven scheduling for cloud video transcoding
Chung-Yiwu, Yu, H.-Y., Huang, J.-C., & Chen, J. (2015) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2015.7340964 · Cited by 4
LS-ADT: Lightweight and Scalable Anomaly Detection for Cloud Datacentres
Barbhuiya, S., Papazachos, Z. C., Kilpatrick, P., & Nikolopoulos, D. S. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-29582-4_8 · Cited by 4
“[2], which extract features from console logs and then use statistical techniques to automatically build models for system anomaly identification.”
BackgroundMethodLopper: An Efficient Method for Online Log Pattern Mining Based on Hybrid Clustering Tree
Liu, J., Hou, Z., & Li, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-27615-7_5 · Cited by 5
Detecting Injection Attacks using Long Short Term Memory
Potinteu, I.-C., & Varga, R. (2020) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp51029.2020.9266177 · Cited by 4
“For training and validation the authors used the HDFS log dataset, that was proposed in [8].”
MethodOn Log Analysis and Stack Trace Use to Improve Program Slicing
Bushong, V., Curtis, J., Sanders, R., Du, M., Cerny, T., Frajták, K., Tisnovsky, P., & Shin, D. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-33-6385-4_25 · Cited by 4
Detecting Unknown Cyber Security Attacks Through System Behavior Analysis
Skopik, F., Wurzenberger, M., & Landauer, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04036-8_5 · Cited by 4
Detection of anomalies in the HDFS dataset
Chnib, M., & Gabsi, W. (2023) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera57763.2023.10197797 · Cited by 4
“For example, Xu et al. [32] employ principal component analysis (PCA), which is a dimension reduction algorithm, to detect potential anomalies in large-scale distributed systems.”
Mahboubi, A., Luong, K., Jarrad, G., Camtepe, S. A., Bewong, M., Bahutair, M., & Pogrebna, G. (2025) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2025.104127 · Cited by 5
Enhancing Reasoning Capacity of SLM Using Cognitive Enhancement
Pan, J., Wong, S. L., Chia, X. W., & Yuan, Y. (2025) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic64266.2025.10920811 · Cited by 6
“It is also an important research topic with practical significance in the field of failure identification [1], [2] and security threat detection [3], [4].”
Log parsers' performance on raw logs from Android devices
Bessa, J. A., Filho, R. M., Souza, G., Barreto, R., & de Freitas, R. (2025) · Journal of Internet Services and Applications · doi:10.5753/jisa.2025.5049 · Cited by 4
Accountability for Service Compliance: A Survey
Yao, J., Chen, S., & Levy, D. (2012) · International Journal of Systems and Service-Oriented Engineering · doi:10.4018/jssoe.2012010102 · Cited by 4
InvarNet-X: A Black-Box Invariant-Based Approach to Diagnosing Big Data Systems
Chen, P., Qi, Y., & Hou, D. (2015) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2015.2497143 · Cited by 4
Wang, R., Ying, S., Sun, C., Wan, H., Huolin, Z., & Jia, X. (2017) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2017-128 · Cited by 4
“The operational information extracted from the running log can help operation and maintenance managers to analyze software status, there are two main means: One is to detect the anomaly of the software by analyzing the log features, for example, Wei Xu[1] proposed an automatic method that extracts information from con…”
MethodFault-tolerance analyzer: A middle layer for pre-provision testing in OpenStack
Hussain, F., Haider, S. A., Alamri, A., & Alqarni, M. A. (2017) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2017.11.019 · Cited by 4
A novel method to analyze logs generated by wireless telecommunication systems
Chen, R., Ji, W., Duan, S., Ling, Q., & Li, F. (2017) · International Conference on Advanced Robotics and Mechatronics (ICARM) · doi:10.1109/icarm.2017.8273166 · Cited by 4
Work in Progress: Topic Modeling for HPC Job State Prediction
DeLucia, A., & Baseman, E. (2018) · doi:10.1145/3217871.3217874 · Cited by 4
Finding Anomalies in Network System Logs with Latent Variables
Otomo, K., Kobayashi, S., Fukuda, K., & Esaki, H. (2018) · doi:10.1145/3229607.3229608 · Cited by 5
“To detect traffic anomalies, Principal Component Analysis (PCA) is a well-studied algorithm [10, 16].”
MethodThe Runtime System Problem Identification Method Based on Log Analysis
Liu, Y., Lv, J., Ma, S., & Yao, W. (2018) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn.2018.8487466 · Cited by 4
“The method proposed by Xu [10] extracts log patterns by analyzing source code and then uses PCA-based algorithm to identify outlier as anomaly.”
MethodSlop: Towards an Efficient and Universal Streaming Log Parser
Zhao, Z., Wang, C., & Rao, W. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01950-1_19 · Cited by 4
Digging Evidence for Violation of Cloud Security Compliance with Knowledge Learned from Logs
Yuan, Y., Torgonshar, A., Shi, W., Liang, B., & Qin, B. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-5913-2_20 · Cited by 6
Improving the system log analysis with language model and semi-supervised classifier
Li, G., Zhu, P., Cao, N., Wu, M., Chen, Z., Cao, G., Li, H., & Gong, C. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-018-7020-3 · Cited by 4
Non-Intrusive Cloud Application Transaction Pattern Discovery
Horovitz, S., Arian, Y., Vaisbrot, M., & Peretz, N. (2019) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2019.00059 · Cited by 4
“The authors in [22], [25] mine workflows in”
BackgroundFinding Performance Patterns from Logs with High Confidence
Kimball, J., Lima, R. A., & Pu, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59618-7_11 · Cited by 4
Valid Probabilistic Anomaly Detection Models for System Logs
Liu, C., Pan, L., Gu, Z., Wang, J., Ren, Y., & Wang, Z. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8827185 · Cited by 5
“[10] proposes to automatically detect system runtime problems by parsing the console system log, and use a PCA-based feature extraction algorithm to accurately describe the complex state information of a large-scale system.”
MethodLeveraging Website Popularity Differences to Identify Performance Anomalies
Grassi, G., Teixeira, R., Barakat, C., & Crovella, M. E. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488832 · Cited by 4
“PCA has been broadly used for anomaly detection, for instance in wide-area communication networks [27, 28], datacenter services [29], and social network activity [30].”
Software Log Anomaly Detection Method Using HTM Algorithm
Hirakawa, R., Tominaga, K., & Nakatoh, Y. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-33-6981-8_6 · Cited by 4
Server-Language Processing: A Semi-Supervised approach to Server Failure Detection
Syngal, S., Verma, S., Karthik, K., Katyal, Y., & Ghosh, S. (2021) · International Conference on Computing, Networks and Internet of Things (CNIOT) · doi:10.1145/3468691.3468716 · Cited by 4
Jian, Z., Jin, Z., Xie, X., Lu, Y., Li, G., Chen, X., & Baker, T. (2021) · Measurement · doi:10.1016/j.measurement.2021.110175 · Cited by 4
A Taxonomy of Anomalies in Log Data
Wittkopp, T., Wiesner, P., Scheinert, D., & Kao, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14135-5_12 · Cited by 8
A Framework for Adaptive Deep Reinforcement Semantic Parsing of Unstructured Data
Jain, S., de Buitléir, A., & Fallon, E. (2021) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc52510.2021.9620904 · Cited by 4
An Approach of Automated Anomalous Microservice Ranking in Cloud-Native Environments
Zhang, Z., Li, B., Wang, J., & Liu, Y. (2021) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194021400167 · Cited by 5
Intelligent Log Analysis Using Machine and Deep Learning
Yen, S., & Moh, M. (2020) · IGI Global eBooks · doi:10.4018/978-1-7998-7705-9.ch053 · Cited by 4
Spray: Streaming Log Parser for Real-Time Analysis
Zou, F., Chen, X., Luo, Y., Huang, T., Zhihong, L., & Song, K. (2022) · Security and Communication Networks · doi:10.1155/2022/1559270 · Cited by 4
“In [25, 26], static analysis techniques are employed to obtain log template information from program source codes.”
MethodLPV: A Log Parsing Framework Based on Vectorization
Xiao, T. Y., Quan, Z., Wang, Z.-J., Zhao, K., Liao, X., Huang, H., Du, Y., & Li, K. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3248124 · Cited by 5
“The second type dives into the source code to reach the corresponding “print” statements of log messages [27], which can gain accurate log templates theoretically, but requires the source code to be available.”
BackgroundMethodInfluentialRobust Log-Based Anomaly Detection with Hierarchical Contrastive Learning
Zhao, Y., Yang, R., Yang, N., Lin, T., Fu, Q., & Ma, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094961 · Cited by 5
“Traditional log-based anomaly detection method-s such as PCA [12], Invariants Mining [13] and some deep methods like DeepLog [14] and logAnomaly [15] require log parsing to obtain log templates/log events as a preprocessing step.”
MethodSNNLog: A Log Parsing Scheme with Siamese Network and Fixed Depth Tree in Networks
Sun, L., & Xu, X. (2023) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd57460.2023.10152745 · Cited by 4
Using Web Server Logs to Identify and Comprehend Anomalous User Activity
Benova, L., & Hudec, L. (2023) · International Conference on Telecommunications (ConTEL) · doi:10.1109/contel58387.2023.10199092 · Cited by 4
“The studied algorithms include statistical modeling (PCA [7], LogCluster [8], and IM [9]) and deep learning-based approaches (DeepLog [10] and LogAnomaly [11]).”
BackgroundAn Anomaly Detection Algorithm for Logs Based on Self-attention Mechanism and BiGRU Model
Yang, H. S., Lin, F., Chai, Y., Qie, K., Lin, W., Wang, Y., Zhang, C., & Guo, M. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-6847-3_76 · Cited by 4
TAElog: A Novel Transformer AutoEncoder-Based Log Anomaly Detection Method
Zhao, C., Huang, K., Wu, D., Han, X., Du, D., Zhou, Y., Lü, Z., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-0945-8_3 · Cited by 4
Analyzing performance traces using temporal formulas
Ryckbosch, F., & Ζ, A. D. (2014) · Software Practice and Experience · doi:10.1002/spe.2256 · Cited by 4
“Xu et al. [35] use data mining techniques to detect performance anomalies using the applications’ source code and console logs.”
MethodLightweight Task Graph Inference for Distributed Applications
Xin, B., Eugster, P. T., Zhang, X., & Yang, J. (2010) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2010.20 · Cited by 4
“[18] developed a general anomaly detection methodology for large-scale systems, in which they studied the effectiveness of applyin g the statistical method PCA, Principal Component Analysis, to feature matricies that were automatically constructed f rom console logs.”
MethodAutoLog: facing log redundancy and insufficiency
Zhang, C., Guo, Z., Wu, M., Lu, L., Yu, F., Zhao, J., & Zhang, Z. (2011) · Asia-Pacific Workshop on Systems · doi:10.1145/2103799.2103811 · Cited by 4
“However, as discussed in prior work [6, 7], a more general log parsing technique is often necessary, since many systems actually have their own logging mechanisms.”
BackgroundDCR: Replay Debugging for the Datacenter
Altekar, G., & Stoica, I. G. (2010) · Cited by 4
Hybrid big data architecture for high-speed log anomaly detection
Tangsatjatham, P., & Nupairoj, N. (2016) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse.2016.7748933 · Cited by 4
Anomaly States Monitoring of Large-Scale Systems with Intellectual Analysis of System Logs
Sheluhin, O. I., & Владимирович, О. А. (2019) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct.2019.8711915 · Cited by 4
“To use the observed unstructured data in machine learning algorithms, it is necessary to carry out their numerical representation [11].”
MethodAnomalies Detection and Proactive Defence of Routers Based on Multiple Information Learning
Li, T., Ma, J., Shen, Y., & Pei, Q. (2019) · Entropy · doi:10.3390/e21080734 · Cited by 4
“The authors of [3,4] needed to combine the source code of the systems to do the analysis.”
An Anomaly Detection Method for System Logs Using Venn-Abers Predictors
Pan, L., Gu, Z., Ren, Y., Liu, C., & Wang, Z. (2020) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc50466.2020.00063 · Cited by 4
“1) are distinguished from the log data content [15-16].”
BackgroundLatent Semantics Approach for Network Log Analysis: Modeling and its application
Otomo, K., Kobayashi, S., Fukuda, K., & Esaki, H. (2021) · IFIP/IEEE International Symposium on Integrated Network Management (IM) · Cited by 6
Padur, S. K. R. (2021) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit217554 · Cited by 4
YTrace: End-to-end Performance Diagnosis in Large Cloud and Content ProvidersPreprint
Kanuparthy, P., Dai, Y.-C., Pathak, S., Samal, S., Benson, T., Ghasemi, M., & Narayan, P. P. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03273 · Cited by 4
Runtime Analysis of Whole-System ProvenancePreprint
Pasquier, T., Han, X. M., Moyer, T., Bates, A., Hermant, O., Eyers, D. M., Bacon, J., & Seltzer, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06049 · Cited by 10
“Timely investigation of system intrusions remains a notoriously difficult challenge [66, 94, 96].”
BackgroundAWSOM-LP: An Effective Log Parsing Technique Using Pattern Recognition and Frequency AnalysisPreprint
Sedki, I., Hamou‐Lhadj, A., & Mohamed, O. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15473 · Cited by 4
“To make things worse, logs are rarely structured [1] [20], making it difficult to extract meaningful information from large raw log files [25] [26].”
BackgroundMethodAnomaly Detection via Mining Numerical Workflow Relations from LogsPreprint
Zhang, B., Zhang, H., Moscato, P., & Zhang, A. (2020) · doi:10.36227/techrxiv.12570926 · Cited by 4
FeadSeq: A Personalized Federated Anomaly Detection Framework for Discrete Event Sequences
Guan, W., Cao, J., Zhao, H., Gu, Y., & Qian, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3742896 · Cited by 4
“System logs: We opt to utilize the HDFS dataset [24, 65] and the BGL (Blue Gene/L) dataset [24, 48] for our analysis.”
GELog: a GPT-Enhanced Log Representation Method for Anomaly Detection
Xu, W., Wang, P., Shi, H., Zhou, G., Yao, J., & Zhang, X. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00062 · Cited by 3
“A. Experimental settings 1) Data: In this paper, we evaluate GELog on four datasets: HDFS [29], Blue Gene/L (BGL), Thunderbird, and Spirit [30].”
Self-citationCSCLog: A Component Subsequence Correlation-Aware Log Anomaly Detection Method
Chen, L., Song, C., Wang, X. A., Fu, D., & Zhou, W. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3585547 · Cited by 3
Li, Z., Yu, J., Huang, Z., & Huang, Y. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3631913 · Cited by 5
Nasirzadeh, M., Tahmoresnezhad, J., & Khazaee, P. R. (2025) · Scientific Reports · doi:10.1038/s41598-025-27693-4 · Cited by 3
“8, 9, 10,11, 12, 13, 14.”
Task-Aware Reduction for Scalable LLM-Database Systems
Barnes, M. E., Ghaleb, T. A., & Hassan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00114 · Cited by 3
“They enable capabilities such as conversational querying, automated debugging, and intelligent system triage, transforming how users interact with large-scale, heterogeneous data [3, 4].”
TransEdge: a lightweight transformer-based anomaly detection framework for edge devices
Shi, Q., Yang, Z., & Shao, S. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01043-w · Cited by 4
“(2) Mixture complexity selection (BIC vs. fixed binary mixture): comparing EM-AT with EM-AT-bin shows that BIC-based selection yields con - sistently equal or higher F 1 (e.g., 92.47% vs. 92.38% on WADI; 98.90% vs. 98.67% on HDFS), supporting BIC as a robust, fully automated default for mixture complexity.”
InfluentialCausal AI for Root Cause Detection in Cloud Process Pipelines
Namdeo, A. (2024) · International Journal of Research and Applied Innovations · doi:10.15662/ijrai.2024.0703010 · Cited by 3
Towards a Training-Oriented Adaptive Decision Guidance and Support System
Zulkernine, F. H., Martin, P., Soltani, S., Powley, W., Mankovskii, S., & Addleman, M. (2012) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew.2012.74 · Cited by 3
“Finally, Section 6 summarizes our contributions and draws the conclusion.”
BackgroundExploiting Data-pattern-aware Vertical Partitioning to Achieve Fast and Low-cost Cloud Log Storage
Wei, J., Zhang, G., Chen, J., Wang, Y., Zheng, W., Sun, T., Wu, J., & Jiang, J. (2024) · ACM Transactions on Storage · doi:10.1145/3643641 · Cited by 3
Easy over Hard: A Simple Baseline for Test Failures Causes Prediction
Gao, Z., Xue, Z., Hu, X., Shang, W., & Xia, X. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663850 · Cited by 3
“In a large software company, thousands of test failures are reported daily resulting into a huge amount of logs, with each log containing hundreds of test steps and thousands of log lines [2, 17, 38].”
BackgroundSruthi, P. C., Guo, Z., Chu, D., Chen, Z., & Zhang, Y. (2024) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3698038.3698568 · Cited by 5
Threat Detection Framework Based on Industrial Internet of Things Logs
Zhang, S., Wang, Y., & Su, X. (2024) · IEEE Access · doi:10.1109/access.2024.3514097 · Cited by 4
CGP-Mamba: An Intrusion Detection Method Based on State Space Model
He, J., Wang, G., Liu, N., & Kong, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4509-1_18 · Cited by 3
Leveraging Deep Learning for Fault Detection and Localization in Distributed Systems
Ghosh, D., & Singh, J. P. (2025) · IEEE Access · doi:10.1109/access.2025.3587529 · Cited by 3
LogAD: A Multi-Feature Fusion Approach for Log Anomaly Detection
Wang, G., Zhang, L., Wang, J., Wo, T., Wang, X., & Hu, C. (2025) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc67032.2025.00016 · Cited by 3
“We evaluate LogAD using four public log datasets widely used in the field of log anomaly detection: HDFS [9], BGL [10], Spirit [10] and Thunderbird [10].”
HAMS: an AI-driven framework for real-time failure detection in HPC system logs
Bang, B., Kim, J., Song, U., Lee, J., & Ma, J. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07849-9 · Cited by 3
“Inspired by the approach described in Drain [43], we used regular expressions for detection and replacement to process the log messages, transforming them as shown in Fig.”
DDC: Distributed Data Collection Framework for Failure Prediction in Tianhe Supercomputers
Hu, W., Jiang, Y., Liu, G., Dong, W., & Cai, G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23216-4_2 · Cited by 3
“The researchers of Rutgers University and IBM company [5–7], Lan research team [8–10], Oliner research team [11–13] and some other researchers [14–17] built failure prediction model based on RAS log data.”
BackgroundDynamic logging with Dylog for networked embedded systems
Dong, W., Huang, C. H., Wang, J., Chen, C., & Bu, J. (2014) · IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/sahcn.2014.6990375 · Cited by 3
Identifying silent failures of SaaS services using finite state machine based invariant analysis
Goel, G., Roy, A. B., & Ganesan, R. (2013) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2013.6688909 · Cited by 3
“Currently, failure diagnosis involves using or mining global invariants [4], [5].”
BackgroundResearch on Cloud-Based Mass Log Data Management Mechanism
Yang, F., Liu, H., & Zhao, Z. (2014) · Journal of Computers · doi:10.4304/jcp.9.6.1371-1377 · Cited by 3
Differentiated Failure Remediation with Action Selection for Resilient Computing
Huang, S., Fu, S., DeBardeleben, N., Guan, Q., & Xu, C. (2015) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2015.42 · Cited by 3
“Many papers have been published to address this issue, such as [41], [35], [38].”
BackgroundExtending Static Code Analysis with Application-Specific Rules by Analyzing Runtime Execution Traces
Ersoy, E., & Sözer, H. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-47217-1_4 · Cited by 3
“There exist a few approaches [9,10,13] that exploit dynamic analysis and runtime execution traces.”
BackgroundThe difference-of-datasets framework: A statistical method to discover insight
Raff, P., & Jin, Z. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840800 · Cited by 3
An Ensemble Signature-Based Approach for Performance Diagnosis in Big Data Platform
Kou, H., & Chen, P. (2018) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2018.00022 · Cited by 3
Wang, R., & Ying, S. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5729 · Cited by 3
Lognroll: discovering accurate log templates by iterative filtering
Tak, B., & Han, W.-S. (2021) · International Middleware Conference · doi:10.1145/3464298.3493400 · Cited by 7
“Other approaches include a technique based on the source-code analysis [37].”
MethodLogCAD: An Efficient and Robust Model for Log-Based Conformal Anomaly Detection
Liu, C., Liang, M., Hou, J., Gu, Z., & Wang, Z. (2022) · Security and Communication Networks · doi:10.1155/2022/5822124 · Cited by 3
Syslog Episode Mining for Tracking Behavior in SCADA servers
Yang, X., Biswas, J., Yau, D. K. Y., & Yu, M. (2022) · doi:10.1109/icostech54296.2022.9829119 · Cited by 3
Supervised Learning for Detecting Cognitive Security Anomalies in Real-Time Log Data
Amin, M. A. (2022) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic55036.2022.9848922 · Cited by 3
“To reduce manual efforts in log parsing, a study [6] examined static analysis algorithms to extract event templates directly from source code.”
MethodA Complexity Aware Scheduler With Dynamic Slot Allocation for Cloud Video Transcoding
Chen, J., Yu, H.-Y., Hwuang, C.-C., Sung, C.-J., & Wu, Y.-S. (2022) · IEEE Access · doi:10.1109/access.2022.3210528 · Cited by 3
“After all t i s have been processed by respective workers, they are transmitted to a reducer to integrate small tasks back to a finished job [4], [5].”
MethodAnomaly Detection from Log Data Sequences with Perturbations
Chen, Y., Ye, L., Ye, Y., Zhang, P., & Tan, Q. (2022) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc55868.2022.00031 · Cited by 3
“Unsupervised methods include principal component analysis (PCA) [15], InvariantsMiner and so on [16], which are able to capture quantitative anomalies of log sequences.”
BackgroundMethodAn Anomaly Detection Approach of Part-of-Speech Log Sequence via Population Based Training
Zhang, X., Zhang, J., Yang, J., Lin, F., Wang, C., Chang, L., & Li, D. (2023) · IEEE International Conference on Power, Electronics and Computer Applications (ICPECA) · doi:10.1109/icpeca56706.2023.10075876 · Cited by 3
Detecting log anomaly using subword attention encoder and probabilistic feature selection
Hariharan, M., Mishra, A., Ravi, S., Sharma, A., Tanwar, A., Sundaresan, K., Ganesan, P., & Karthik, R. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04674-6 · Cited by 3
“OpenStack is a set of software components to manage cloud services and infrastructure [36].”
MethodA Robust Log Classification Approach Based on Natural Language Processing
Li, D., Zhang, J., Yang, J., Zhang, X., Lin, F., Wang, C., & Chang, L. (2023) · International Conference on Computer, Control and Robotics (ICCCR) · doi:10.1109/icccr56747.2023.10194171 · Cited by 3
HCLPars: Α New Hierarchical Clustering Log Parsing Method
Binlashram, A., Hsairi, L., & Ahmadi, H. A. (2023) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.6013 · Cited by 3
“For example, authors in [13] proposed a general methodology for log analysis based on source code analysis to discover largescale system issues by extracting log events from console logs.”
BackgroundMethodTowards Massive Log Anomaly Detection Based on an Enhanced Multi-dimensional Time-domain LSTM
Wang, Y., Zhang, C., & Yu, J. (2023) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice58306.2023.00042 · Cited by 3
Ensemble Deep Learning based Real-time Log Anomaly Detection
Puranik, A., Akkihal, A. V., Suhas, R. K., Patel, Y. P. D., & Mahesh, H. B. (2023) · International Conference on Digital Applications, Transformation & Economy (ICDATE) · doi:10.1109/icdate58146.2023.10248821 · Cited by 3
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.