Detecting large-scale system problems by mining console logs
The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 4
- [2] D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007. -
- [5] R. Dunia and S. J. Qin. Multi-dimensional fault diagnosis using a subspace approach. In Proc. ACC , 1997. -
- [8] R. Fonseca and et al. Xtrace: A pervasive network tracing framework. In In Proc. NSDI , 2007. -
- [14] W. Jiang and et al. Understanding customer problem troubleshooting from storage system logs. In Proceedings of USENIX FAST'09 , 2009. -
Cited by
Items 301–400 of 1,004
Pcatch: automatically detecting performance cascading bugs in cloud systems
Li, J., Chen, Y., Liu, H., Lu, S., Zhang, Y., Gunawi, H. S., Gu, X., Lu, X., & Li, D. (2018) · European Conference on Computer Systems · doi:10.1145/3190508.3190552 · Cited by 32
“Many tools have been built to diagnose system performance anomalies [4, 6, 11, 37–39, 43], including many built for diagnosing and reasoning about performance problems in distributed systems [2, 7, 27, 40, 47, 54].”
BackgroundMultivariate Log-based Anomaly Detection for Distributed Database
Zhang, L., Jia, T., Jia, M., Li, Y., Yang, Y., & Wu, Z. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671725 · Cited by 47
Heading off correlated failures through independence-as-a-service
Zhai, E., Chen, R., Wolinsky, D. I., & Ford, B. (2014) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5555/2685048.2685073 · Cited by 59
CloudRaid: hunting concurrency bugs in the cloud via log-mining
Lu, J., Li, F., Li, L., & Feng, X. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236071 · Cited by 28
“Similar to [48, 52], we represent logging patterns of static messages as regular expressions.”
BackgroundMethodInfluentialEvaluating Scalability Bottlenecks by Workload Extrapolation
Shi, R., Gan, Y., & Wang, Y. (2018) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2018.00039 · Cited by 38
“Log analysis: Log analysis has been used extensively in previous works to infer causal relationship between events [14, 36, 65], to understand system behavior [6, 23, 25, 62, 66], and to debug correctness and performance problems [12, 41, 49, 60, 63].”
MethodAnomaly detection in a forensic timeline with deep autoencoders
Studiawan, H., & Sohel, F. A. (2021) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2021.103002 · Cited by 31
A2Log: Attentive Augmented Log Anomaly Detection
Wittkopp, T., Acker, A., Nedelkoski, S., Bogatinovski, J., Scheinert, D., Wu, F., & Kao, O. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.234 · Cited by 38
Anomaly Detection from Log Files Using Unsupervised Deep Learning
Buršić, S., Cuculo, V., & D’Amelio, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-54994-7_15 · Cited by 30
NetCheck: network diagnoses from blackbox traces
Zhuang, Y., Gessiou, E., Portzer, S., Fund, F., Muhammad, M., Beschastnikh, I., & Cappos, J. (2014) · Networked Systems Design and Implementation · doi:10.5555/2616448.2616460 · Cited by 30
Shang, W., Jiang, Z. M., Adams, B., Hassan, A. E., Godfrey, M. W., Nasser, M., & Flora, P. (2011) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2011.48 · Cited by 30
Research on Log Anomaly Detection Based on Sentence-BERT
Hu, C., Xuekui, S., Dai, H., Zhang, H., & Liu, H. (2023) · Electronics · doi:10.3390/electronics12173580 · Cited by 32
An Approach to Recommendation of Verbosity Log Levels Based on Logging Intention
Anu, H., Chen, J., Shi, W., Hou, J., Liang, B., & Qin, B. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00022 · Cited by 29
“[1] utilized the features extracted from the console logs to detect an anomaly.”
BackgroundMethodLogGAN: A Sequence-Based Generative Adversarial Network for Anomaly Detection Based on System Logs
Xia, B., Yin, J., Xu, J., & Li, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34637-9_5 · Cited by 29
“– PCA [19]: is an unsupervised principal component analysis-based anomaly detection technique which improves the parser of unstructured systems and visualizes the promising diagnosis of abnormal instances.”
MethodInfluentialBlack-box attacks against log anomaly detection with adversarial examples
Lu, S., Wang, M., Wang, D., Wei, X., Xiao, S., Wang, Z., Han, N., & Wang, L. (2022) · Information Sciences · doi:10.1016/j.ins.2022.11.007 · Cited by 32
A feature-based trust sequence classification algorithm
Yahyaoui, H., & Al-Mutairi, A. (2015) · Information Sciences · doi:10.1016/j.ins.2015.08.008 · Cited by 31
“To help in system design, development, evaluation and improvement, machine learning techniques were employed in extracting data from console logs and error reports to describe system behavior and analyze it [7,26,84].”
MethodMining temporal invariants from partially ordered logs
Beschastnikh, I., Brun, Y., Ernst, M. D., Krishnamurthy, A., & Anderson, T. E. (2011) · doi:10.1145/2038633.2038636 · Cited by 29
TaskInsight: A Fine-Grained Performance Anomaly Detection and Problem Locating System
Zhang, X., Meng, F., Chen, P., & Xu, J. (2016) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2016.0136 · Cited by 29
An Adaptive and Memory Efficient Sampling Mechanism for Partitioning in MapReduce
Slagter, K. D., Hsu, C., & Chung, Y. (2013) · International Journal of Parallel Programming · doi:10.1007/s10766-013-0288-z · Cited by 28
“Therefore there has been a variety of research on MapReduce in order to improve the performance of the framework or the performance of specific applications like datamining [20,28], graph mining [10], text analysis [25] or genetic algorithms [11,26] that run on the framework.”
BackgroundvNetTracer: Efficient and Programmable Packet Tracing in Virtualized Networks
Suo, K., Zhao, Y. B., Chen, W., & Rao, J. (2018) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2018.00026 · Cited by 28
“, data mining, machine learning [28], [43], [51], [57], [56], are promising directions towards automated identification of the issues inside the systems based on the massive tracing logs, the execution environment as well as the application performance.”
BackgroundDecentralized Federated Learning Preserves Model and Data Privacy
Wittkopp, T., & Acker, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-76352-7_20 · Cited by 33
LogSummary: Unstructured Log Summarization for Software Systems
Meng, W., Zaiter, F., Zhang, Y., Liu, Y., Zhang, S., Tao, S., Zhu, Y., Han, T., Zhao, Y., Wang, E., et al. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3236994 · Cited by 32
ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection
Egersdoerfer, C., Zhang, D., & Dai, D. (2022) · IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS) · doi:10.1109/ftxs56515.2022.00006 · Cited by 28
“As a result, an extensive amount of research on log-based anomaly detection has been done recently [8], [26], [16], [25], [34], [10], [7], [19], [12], [22], [32], [33], [21], [20], [9].”
BackgroundBridging the gaps: joining information sources with Splunk
Stearley, J., Corwell, S. E., & Lord, K. (2010) · Cited by 28
Detecting Large-Scale System Problems by Mining Console Logs
Xu, W., & Huang, L. (2010) · Cited by 28
Self-citationGabel, M., Schuster, A., & Keren, D. (2014) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2014.16 · Cited by 27
“Like the original framework, our adaptation admits different comparison functions, supports non-stationary data, and provides statistical guarantees on the rate of false positives.”
BackgroundA Survey On Log Research Of AIOps: Methods and Trends
Jiang, Z., Tong, L., Zhenguo, Z., Ge, J., Junling, Y., & Li, L. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-021-01832-3 · Cited by 40
“There are also some methods of source code analysis, as mentioned in [27, 28], which face the challenges of cross-platform, cross-languages, and unreachable.”
MethodA Semisupervised Approach for Industrial Anomaly Detection via Self-Adaptive Clustering
Ma, X., Keung, J., He, P., Xiao, Y., Yu, X., & Li, Y. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3280246 · Cited by 27
“We evaluate the performance of AdaLog with comprehensive experiments on three widely used log anomaly datasets (i.e., HDFS [13], Blue Gene/L supercomputer (BGL) [18], and Thun-derbird [18]).”
BackgroundMethodStudying logging practice in machine learning-based applications
Foalem, P. L., Khomh, F., & Li, H. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107450 · Cited by 34
Relational Synthesis of Text and Numeric Data for Anomaly Detection on Computing System Logs
Baseman, E., Blanchard, S. P., Zongze, L., & Fu, S. (2016) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2016.0158 · Cited by 27
An unsupervised framework for detecting anomalous messages from syslog log files
Vaarandi, R., Blumbergs, B., & Kont, M. (2018) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2018.8406283 · Cited by 27
“For addressing this issue, various anomaly detection methods have been suggested, including hidden Markov models [3, 4], principal component analysis (PCA) [5], decision trees [6], entropy based algorithms [7, 8], support vector machines (SVM) [9, 10], neural networks [11], and logistic regression [12].”
BackgroundMethodInfluentialLogSayer: Log Pattern-driven Cloud Component Anomaly Diagnosis with Machine Learning
Zhou, P., Wang, Y., Li, Z., Wang, X., Gareth Tyson, A., & Xie, G. (2020) · International Workshop on Quality of Service · doi:10.1109/iwqos49365.2020.9212954 · Cited by 31
“We choose three representative approaches in cloud system anomaly diagnosis to compare with the LogSayer: • PCA approach [30], which divides logs to different ’sessions’, assuming that there is a session ID in the log message.”
BackgroundMethodInfluentialTrustworthy AI-based Performance Diagnosis Systems for Cloud Applications: A Review
Xin, R., Wang, J., Chen, P., & Zhao, Z. (2025) · ACM Computing Surveys · doi:10.1145/3701740 · Cited by 30
“Figure 4(a) shows an example of log data from the HDFS logs on the Amazon EC2 platform [210].”
LEMAD: LLM-Empowered Multi-Agent System for Anomaly Detection in Power Grid Services
Ji, X., Zhang, L., Zhang, W., Peng, F., Mao, Y., Liao, X., & Zhang, K. (2025) · Electronics · doi:10.3390/electronics14153008 · Cited by 33
Demystifying the challenges and benefits of analyzing user-reported logs in bug reports
Chen, A. R., Chen, T.-H. P., & Wang, S. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09893-w · Cited by 37
“Other studies apply machine learning techniques to identify anomalies in the log messages, which may be an indication of possible problems (Xu et al. 2009; Chen et al. 2017a; Lin et al. 2016).”
BackgroundConAnomaly: Content-Based Anomaly Detection for System Logs
Lv, D., Luktarhan, N., & Chen, Y. (2021) · Sensors · doi:10.3390/s21186125 · Cited by 32
“Existing log-based system anomaly detection methods can be roughly classified into two categories: one is based on log event indexes, such as PCA [11], Invariant Mining [12], Deeplog [13], and QLLog [14].”
MethodInfluentialUSTEP: Unfixed Search Tree for Efficient Log Parsing
Vervaet, A., Chiky, R., & Callau-Zori, M. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00077 · Cited by 31
ConfSeer: leveraging customer support knowledge bases for automated misconfiguration detection
Potharaju, R., Chan, J., Hu, L., Nita-Rotaru, C., Wang, M., Zhang, L., & Jain, N. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824079 · Cited by 26
“Recently, there have been proposals [70, 42, 69, 76] on detecting anomalous patterns based on console logs or ground truth configuration snapshots.”
BackgroundBehavioral anomaly detection approach based on log monitoring
Du, S., & Cao, J. (2015) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc.2015.7365981 · Cited by 26
Improving the Software Logging Practices in DevOps
Chen, B. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00080 · Cited by 26
“[2] proposed an anomaly detection technique to flag problematic behavior”
MethodLogM : Log Analysis for Multiple Components of Hadoop Platform
Xie, Y., Yang, K., & Luo, P. (2021) · IEEE Access · doi:10.1109/access.2021.3076897 · Cited by 29
“Besides, logs with common variables over a period of time possibly come from the same event [18].”
BackgroundSentiLog: Anomaly Detecting on Parallel File Systems via Log-based Sentiment Analysis
Zhang, D., Dai, D., Han, R., & Zheng, M. (2021) · ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) · doi:10.1145/3465332.3470873 · Cited by 31
“An extensive number of methods have been proposed [15, 18, 19, 22, 23, 27, 35–38, 40, 44, 49–51], which can be roughly classified into three categories: (a) rule-based methods that rely on rules summarized by experts to detect the anomalies, such as LogLens [22] and PerfAugur [44]; (b) supervised learning methods that…”
BackgroundMethodSemi-supervised log anomaly detection based on bidirectional temporal convolution network
Yin, Z., Kong, X., & Yin, C. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103808 · Cited by 25
LogEDL: Log Anomaly Detection via Evidential Deep Learning
Duan, Y., Xue, K., Sun, H., Bao, H., Wei, Y., You, Z., Zhang, Y., Jiang, X., Yang, S., Chen, J., et al. (2024) · Applied Sciences · doi:10.3390/app14167055 · Cited by 32
Stage-aware anomaly detection through tracking log points
Ghanbari, S., Hashemi, A. B., & Amza, C. (2014) · International Middleware Conference · doi:10.1145/2663165.2663319 · Cited by 25
“In summary, currently users have only two choices: i) to forgo DEBUG-level logging in production systems and give up the crucial information that those logs provide for problem diagnosis, or ii) to pay the high costs of DEBUG-level logging in conjunction with ad-hoc, approximate, and error-prone log mining solutions […”
BackgroundMethodInfluentialPerformance Issue Diagnosis for Online Service Systems
Fu, Q., Lou, J., Lin, Q., Ding, R. F., Zhang, D., Ye, Z., & Xie, T. (2012) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2012.49 · Cited by 25
DReAM: Deep Recursive Attentive Model for Anomaly Detection in Kernel Events
Ezeme, O. M., Mahmoud, Q. H., & Azim, A. (2019) · IEEE Access · doi:10.1109/access.2019.2897122 · Cited by 29
Anchorage: Visual Analysis of Satisfaction in Customer Service Videos Via Anchor Events
Kam-Kwai, W., Wang, X., Wang, Y., He, J., Zhang, R., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3245609 · Cited by 25
“The Principle Component Analysis (PCA) [62] is a popular unsupervised method for system log analysis detecting anomalous discrete events.”
MethodMining causality graph for automatic web-based service diagnosis
Nie, X., Zhao, Y., Sui, K., Pei, D., Chen, Y., & Qu, X. (2016) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/pccc.2016.7820614 · Cited by 30
“Some methods are based on mining log [18–20].”
MethodAn online log template extraction method based on hierarchical clustering
Yang, R., Qu, D., Qian, Y., Dai, Y., & Zhu, S. (2019) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-019-1430-4 · Cited by 28
Iturbe, M., Garitano, I., Zurutuza, U., & Uribeetxeberria, R. (2017) · Security and Communication Networks · doi:10.1155/2017/9150965 · Cited by 28
“[42] is an ADS based on host log mining.”
BackgroundBridging the divide between software developers and operators using logs
Shang, W. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227031 · Cited by 25
Experiences with Tracing Causality in Networked Services
Fonseca, R., Freedman, M. J., & Porter, G. (2010) · Cited by 25
OneLog: towards end-to-end software log anomaly detection
Hashemi, S., & Mäntylä, M. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00428-x · Cited by 24
“After removing redundancies, the dataset remained with 6,453,814 normal and 5,114 anomalous samples.”
Ding, Z., Tang, Y., Cheng, X., Li, H., & Shang, W. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3624740 · Cited by 31
“Prior work has leveraged the rich information in logs to support diferent software engineering activities, including system comprehension [20, 71], anomaly detection [21, 33, 48, 55, 88, 89], and failure diagnosis [61, 73].”
BackgroundLeveraging Large Language Models and BERT for Log Parsing and Anomaly Detection
Zhou, Y., Chen, Y., Rao, X., Zhou, Y., Li, Y., & Hu, C. (2024) · Mathematics · doi:10.3390/math12172758 · Cited by 32
Nie, J., Liu, G., Li, C., & Zou, T. (2026) · International Journal of Graphic Design · doi:10.51903/ijgd.v4i1.3703 · Cited by 24
Constructing the Knowledge Base for Cognitive IT Service Management
Wang, Q., Zhou, W., Zeng, C., Li, T., Shwartz, L., & Grabarnik, G. Y. (2017) · IEEE International Conference on Services Computing · doi:10.1109/scc.2017.59 · Cited by 24
Mithra: Anomaly Detection as an Oracle for Cyberphysical Systems
Afzal, A., Goues, C. L., & Timperley, C. S. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3120680 · Cited by 25
“Other techniques use console logs generated by the system as the source for mining invariants and detecting anomalies [14], [67], [102], [113].”
Wang, Q., Zhang, X., Wang, X., & Cao, Z. (2021) · Entropy · doi:10.3390/e24010069 · Cited by 29
“It then uses the unsupervised method of PCA to mine the feature vectors, and mark each feature vector as normal or abnormal, but this method lacks focus only on the text itself, and lacks the analysis and utilization of the semantics of the text, which affects the accuracy of anomaly detection.”
BackgroundMethodInfluentialAdanomaly: Adaptive Anomaly Detection for System Logs with Adversarial Learning
Qi, J., Luan, Z., Huang, S., Wang, Y., Fung, C., Yang, H., & Qian, D. (2022) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms54207.2022.9789917 · Cited by 26
“For example, [2] used the PCA finding patterns in the dimension of the event count vector to generate abnormal space and normal space.”
MethodLin, Y., Wang, Z., Lin, P., Nguyen, V.-L., Hwang, R., & Lai, Y. (2022) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2022.103248 · Cited by 37
“For example, Xu et al. [21] uses principal component analysis to analyze feature vectors.”
MethodGoing through the Life Cycle of Faults in Clouds: Guidelines on Fault Handling
Li, X., Yu, G., Chen, P., Chen, H., & Chen, Z. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00022 · Cited by 26
“With the prevalence of machine learning, data-driven fault detection methods such as [50], [51] attract much attention but they remain largely in the realm of research.”
BackgroundDistributed system anomaly detection using deep learning‐based log analysis
Han, P., Li, H., Xue, G., & Zhang, C. (2023) · Computational Intelligence · doi:10.1111/coin.12573 · Cited by 27
ADP: automated diagnosis of performance pathologies using hardware events
Yoo, W., Larson, K., Baugh, L., Kim, S. K., & Campbell, R. H. (2012) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/2254756.2254791 · Cited by 29
Discovering process models for the analysis of application failures under uncertainty of event logs
Pecchia, A., Weber, I. M., Cinque, M., & Ma, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105054 · Cited by 24
AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges
Cheng, Q., Sahoo, D., Saha, A., Yang, W., Liu, C., Woo, G., Singh, M., Saverese, S., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2304.04661 · Cited by 23
“For tackling this, some of the more traditional statistical methods like [103], [95], [82], [84] support online streaming update while some other works can also adapt to evolving”
MethodInfluentialComprehensive Analysis and Evaluation of Anomalous User Activity in Web Server Logs
Benova, L., & Hudec, L. (2024) · Sensors · doi:10.3390/s24030746 · Cited by 27
“Prior research in log analysis [12,14,19,20] has excluded timestamp values from log entries, focusing exclusively on log keys for identifying anomalies.”
MethodXiao, R., Li, W., Lu, J., & Jin, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3400283 · Cited by 23
“Many automated log-based anomaly detection methods have been proposed [4], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16] in recent years.”
MethodInfluentialAnomaly Detection in Logs Using Deep Learning
Aziz, A., & Munir, K. (2024) · IEEE Access · doi:10.1109/access.2024.3506332 · Cited by 23
“We compare the proposed technique ‘‘Anylog’’ with the four unsupervised existing methods namely LogCluster [21], PCA [40], Invariant Mining (IM) [41], and Deeplog [4].”
Detecting Broken Object-Level Authorization Vulnerabilities in Database-Backed Applications
Huang, Y., Shi, C., Lu, J., Li, H., Meng, H., & Li, L. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690227 · Cited by 23
“[49, 54] to recursively analyze variables within these statements, aiming to distinguish static text from variables in SQL statements.”
LogLLaMA: Transformer-based log anomaly detection with LLaMA
Yang, Z., & Harris, I. G. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227209 · Cited by 23
“Most of such methods are either unsupervised such as Principal Component Analysis (PCA) [2], iForest [3], and LogCluster [4], or semi-supervised such as One-Class Support Vector Machine [5].”
InfluentialRosenberg, C. M., & Moonen, L. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410684 · Cited by 23
“To this end, several authors investigate the combination of log analysis with (static) source code analysis, where they try to (partially) match events in logs to log statements in the code, and then use these statements to reconstruct a path through the source code to help determine what happened in a failed executio…”
BackgroundA Review of Unstructured Data Analysis and Parsing Methods
Jain, S., de Buitléir, A., & Fallon, E. (2020) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci48226.2020.9167588 · Cited by 28
“These unstructured log messages can be processed and parsed to be made available for anomaly detection [2] [3] [4], cognitive management [5], autonomous problem validation [6] [7], incident management or root cause analysis.”
BackgroundDeepTriage: Automated Transfer Assistance for Incidents in Cloud Services
Pham, P. T., Jain, V., Dauterman, L., Ormont, J., & Jain, N. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403380 · Cited by 32
“While prior work focused on fast incident detection [6, 13, 14, 33], there is little research on automating the incident assignment task [10].”
BackgroundZhang, B., Zhang, H., Le, V.-H., Moscato, P., & Zhang, A. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00370-w · Cited by 24
A Survey of Deep Anomaly Detection for System Logs
Zhao, X., Jiang, Z., & Ma, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892726 · Cited by 29
“Labeled data are not available in some systems, so Xu et al. [2] proposed an unsupervised anomaly detection model based on principal component analysis.”
BackgroundMethodInfluentialBhusal, D., Shin, H. J., Shewale, A. A., Veerabhadran, M. K., Clifford, M., Rampazzi, S., & Rastogi, N. (2023) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3600160.3600193 · Cited by 36
“The HDFS dataset comprises of logs generated from running map-reduce jobs on 200 Amazon EC2 nodes and comprises 11.2 million log entries, with 2.9% labeled as anomalies.”
MethodInfluentialLog Anomaly Detection by Adversarial Autoencoders With Graph Feature Fusion
Xie, Y., & Yang, K. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3305376 · Cited by 26
“PCA performs dimension reduction of log features, while FPM mines anomalous patterns within the reduced feature space [13].”
MethodAutoLog: A Log Sequence Synthesis Framework for Anomaly Detection
Huo, Y., Li, Y., Su, Y., He, P., Xie, Z., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00133 · Cited by 30
“In particular, D -HDFS [18], D - Hadoop [17], and D -BGL [21] datasets are collected for anomaly detection after running normal workloads and injecting several types of failures in each system, respectively.”
BackgroundMethodTemporal data mining approaches for sustainable chiller management in data centers
Patnaik, D., Marwah, M., Sharma, R. K., & Ramakrishnan, N. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1989734.1989738 · Cited by 23
PADLA: A Dynamic Log Level Adapter Using Online Phase Detection
Mizouchi, T., Shimari, K., Ishio, T., & Inoue, K. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00029 · Cited by 23
“The information is often used in failure diagnosis [10], [11], [17], program comprehension [15], and so on.”
BackgroundEfficient and Robust Syslog Parsing for Network Devices in Datacenter Networks
Zhang, S., Liu, Y., Meng, W., Bu, J., Yang, S., Sun, Y., Pei, D., Xu, J. J., Zhang, Y., Song, L., et al. (2020) · IEEE Access · doi:10.1109/access.2020.2972691 · Cited by 25
“Specifically, HDFS logs are collected from a node cluster of Amazon EC2 platform with 203 server nodes [38].”
MethodContext-Aware Learning for Anomaly Detection with Imbalanced Log Data
Sun, P., Yuepeng, E., Li, T., Wu, Y., Ge, J., You, J., & Wu, B. (2020) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-smartcity-dss50907.2020.00055 · Cited by 26
“There are three open log datasets: BGL dataset [15], HDFS dataset [16] and Thunderbird dataset [15].”
MethodMADMM: microservice system anomaly detection via multi-modal data and multi-feature extraction
Wang, P., Zhang, X., Cao, Z., & Chen, Z. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09918-1 · Cited by 25
“Xu et al. [20] adopted the log event indices as input to construct the normal and abnormal spaces of the event count vector using the principal component analysis(PCA) algorithm.”
Real-time failure prediction in online services
Shatnawi, M. Q., & Hefeeda, M. M. (2015) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2015.7218516 · Cited by 22
“Furthermore, data mining techniques used to generate failure predictors require high volumes of data to reach high prediction accuracy [5], [7], [3].”
BackgroundMethodInfluentialA new distributed anomaly detection approach for log IDS management based on deep learning
Koca, M., Aydın, M. A., Sertbaş, A., & Zaim, A. H. (2021) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-2102-89 · Cited by 25
LogST: Log Semi-supervised Anomaly Detection Based on Sentence-BERT
Zhang, M., Chen, J., Liu, J., Wang, J., Shi, R., & Hua, S. (2022) · 2022 7th International Conference on Signal and Image Processing (ICSIP) · doi:10.1109/icsip55141.2022.9886069 · Cited by 22
“LogNADS [5] proposes a new semantics representation method of log events, which only selects some words in the log event as subject words, and connects the embedded subject word vectors in word order as the semantic representation of the log event.”
MethodAllInfoLog: Robust Diverse Anomalies Detection Based on All Log Features
Xiao, R., Chen, H., Lu, J., Li, W., & Jin, S. (2022) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3224974 · Cited by 28
“We evaluate AllInfoLog on three kinds of datasets: realworld log datasets [18], [28], synthetic log datasets, and unstable log datasets [13].”
MethodInfluentialStudying the characteristics of AIOps projects on GitHub
Aghili, R., Li, H., & Khomh, F. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10382-z · Cited by 26
GLAD: Content-Aware Dynamic Graphs For Log Anomaly Detection
Li, Y., Liu, Y., Wang, H., Chen, Z., Cheng, W. D., Chen, Y., Yu, W., Chen, H., & Liu, C. (2023) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/ickg59574.2023.00007 · Cited by 28
“• Principal Component Analysis (PCA) [11].”
BackgroundMethodInfluentialBatoun, M. A., Sayagh, M., Aghili, R., Ouni, A., & Li, H. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10452-w · Cited by 31
G 2 : a graph processing system for diagnosing distributed systems
Guo, Z., Zhou, D., Lin, H., Yang, M., Long, F., Deng, C., Liu, C., & Zhou, L. (2011) · Cited by 22
Latent fault detection in large scale services
Gabel, M., Schuster, A., Gilad-Bachrach, R., & Bjørner, N. (2012) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2012.6263932 · Cited by 22
“Moreover, different services, or even different versions of the same service, may have different operating points.”
BackgroundDesign and Development of AD-CGAN: Conditional Generative Adversarial Networks for Anomaly Detection
Ezeme, O. M., Mahmoud, Q. H., & Azim, A. (2020) · IEEE Access · doi:10.1109/access.2020.3025530 · Cited by 23
Automating datacenter operations using machine learning
Patterson, D. A., Jordan, M. I., & Bodík, P. (2010) · UC Berkeley · Cited by 25
Self-citationPractitioners’ expectations on log anomaly detection
Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., & Barr, E. T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586700 · Cited by 21
“Most of the papers evaluate their techniques on two public datasets, Hadoop Distributed File System (HDFS) [27] and Blue Gene/L supercomputer (BGL) [28], typically grouping logs into sequences using sessions and window sizes.”
Explainable LSTM Model for Anomaly Detection in HDFS Log File using Layerwise Relevance Propagation
Patil, A., Wadekar, A., Gupta, T., Vijan, R., & Kazi, F. (2019) · 2019 IEEE Bombay Section Signature Conference (IBSSC) · doi:10.1109/ibssc47189.2019.8973044 · Cited by 21
“As explored by [10], the anomalous log entries are not restricted to only one cause and are of varied types.”
BackgroundConfidence guided anomaly detection model for anti-concept drift in dynamic logs
Xie, X., Jin, Z., Wang, J., Yang, L., Lu, Y., & Li, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102659 · Cited by 22
Identifying and Generating Missing Tests using Machine Learning on Execution Traces
Utting, M., Legeard, B., Dadeau, F., Tamagnan, F., & Bouquet, F. (2020) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest49225.2020.00020 · Cited by 21
“Xu et al. [8] use Principal Component Analysis (PCA) to project traces into a reduced space and thus proceed to anomaly detection, analyzing correlations between traces within a sequence.”
MethodInterpretable Federated Transformer Log Learning for Cloud Threat Forensics
Parra, G. D. L. T., Selvera, L., Khoury, J. D., Irizarry, H., Bou‐Harb, E., & Rad, P. (2022) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2022.23102 · Cited by 30
“Such methods include Principal Component Analysis (PCA) [56], Support Vector Machine (SVM) [20], Bi-LSTM [59], LSTM [16, 38], and Transformers [22, 53].”
Method
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.