The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Detecting large-scale system problems by mining console logs
Abstract
Surprisingly, console logs rarely help operators detect problems in large-scale datacenter services, for they often consist of the voluminous intermixing of messages from many software components written by independent developers. We propose a general methodology to mine this rich source of information to automatically detect system runtime problems. We first parse console logs by combining source code analysis with information retrieval to create composite features. We then analyze these features using machine learning to detect operational problems. We show that our method enables analyses that are impossible with previous methods because of its superior ability to create sophisticated features. We also show how to distill the results of our analysis to an operator-friendly one-page decision tree showing the critical messages associated with the detected problems. We validate our approach using the Darkstar online game server and the Hadoop File System, where we detect numerous real problems with high accuracy and few false positives. In the Hadoop case, we are able to analyze 24 million lines of console logs in 3 minutes. Our methodology works on textual console logs of any size and requires no changes to the service software, no human input, and no knowledge of the software's internals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [49]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“We have investigated a variety of such methods and have found that Principal Component Analysis (PCA) [5, 16] combined with term-weighting techniques from information retrieval [23, 26] yields excellent anomaly detection results on both feature matrices, while requiring little parameter tuning.”
MethodWitten, I. H. (2008) · Cited by 9,143
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
The text mining handbook: advanced approaches in analyzing unstructured data
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5684 · Cited by 1,598
Control Procedures for Residuals Associated With Principal Component Analysis
Jackson, J. E., & Mudholkar, G. S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489779 · Cited by 1,237
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
YALE: rapid prototyping for complex data mining tasks
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150531 · Cited by 1,010
Pinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835 · cited 1×
“, 2002; Ma & Hellerstein, 2001) to more detailed tracing such as execution paths (Chen et al., 2002; Chen & Brewer, 2004).”
BackgroundSelf-citation
Research works printed but not linked: 16
- Borthakur D., 2007, Hadoop Project Website
- Chen M.Y., 2004, Proc. NSDI'04, 23
- Dunia R., 1997, Proc. ACC
- Fonseca R., 2007, In Proc. NSDI
- Hansen S.E., 1993, Proc. USENIX LISA '93, 145
- Hatcher E., 2004, Manning Publications Co.
- Jiang W., 2009, Proceedings of USENIX FAST'09
- Jolliffe I., 2002, Springer
- Lim C., 2008, Proc. DSN
- Ma S., 2001, Proc. IEEE ICDE
- 10.5555/1394399 · doi:10.5555/1394399
- Prewett J.E., 2003, Proc. Annual Conf. on Linux Clusters
- Salton G., 1987, Cornell
- 10.5555/1111682.1111732 · doi:10.5555/1111682.1111732
- Tan J., 2008, Proc. of WASL '08
- Witten I.H., 2000, Morgan Kaufmann
Cited by
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
“Principal Component Analysis (PCA) method [39] assumes that there are different “sessions” in a log file that can be easily identified by a session id attached to each log entry.”
BackgroundMethodInfluentialDrain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodResultInfluentialCondie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
“MapReduce is often used to analyze streams of constantly-arriving data, such as URL access logs [6] and system console logs [28].”
MethodExperience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R. (2016) · 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2016.21 · Cited by 593
“Fortunately, by exploring an abundance of literature and intensively contacting the corresponding authors, we have successfully obtained two log datasets, HDFS data [47] and BGL data [36], which are suitable for evaluating existing anomaly detection methods.”
BackgroundMethodInfluentialTools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
“Typical applications include anomaly detection [12], [18], [23], problem diagnosis [4], [5], program verification [55], [56], performance modeling [3], etc.”
BackgroundMethodInfluentialLogBERT: Log Anomaly Detection via BERT
Guo, H., Yuan, S., & Wu, X. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534113 · Cited by 454
“Regarding baselines, we leverage the package Loglizer [4] to evaluate PCA, OCSVM, iForest as well as LogCluster for anomaly detection and adopt the open source deep learning-based log analysis toolkit to evaluate DeepLog and LogAnomaly 3 .”
InfluentialLog clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J.-G., Zhang, Y., & Chen, X. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2889232 · Cited by 427
“To facilitate analysis, a common practice is to parse the log messages into constant strings and parameters [6, 11, 12, 24] and form abstract log messages.”
BackgroundMethodSpell: Streaming Parsing of System Event Logs
Du, M., & Li, F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0103 · Cited by 423
“To alleviate the pain of diving into massive unstructured log data, in most prior work, the first and foremost step is to automatically parse the unstructured system logs to structured data [2], [3], [4], [6].”
BackgroundMethodInfluentialLog-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“HDFS dataset [34], [7] contains 11,175,629 log messages collected from a Hadoop Distributed File System on the Amazon EC2 platform.”
MethodDeveloping a predictive model of quality of experience for internet video
Balachandran, A., Sekar, V., Akella, A., Seshan, S., Stoica, I., & Zhang, H. (2013) · Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM · doi:10.1145/2486001.2486025 · Cited by 349
“Decision trees can be directly mapped into event processing rules that system designers are typically familiar with [39].”
BackgroundA Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“en-source implementation of existing log parsers, this survey focuses on the methodological comparison of these parsers. In additional, while parsing rules can be constructed by analyzing source code [144], we focus on parsers that only utilize logs as input. Mode. Mode is the most important log parsing characteristic…”
BackgroundMethodInfluentialUnderstanding and detecting real-world performance bugs
Jin, G., Song, L., Shi, X., Scherpelz, J., & Lu, S. (2012) · Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2254064.2254075 · Cited by 304
“Many previous works tried to isolate bottlenecks in (semi-) blackbox systems [1], correlate system metrics with performance behaviors [8, 48], avoid performance problems at runtime [52], and diagnose problems through logs [59].”
BackgroundLoghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302
“The logs collected from the work [73] are sliced into traces (i.”
MethodMining invariants from console logs for system problem detection
Lou, J., Fu, Q., Yang, S., Xu, Y., & Li, J. (2010) · Cited by 302
Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00130 · Cited by 244
“3: Visualization of main stages in HDBSCAN clustering by sampling 100 log sequences in the HDFS dataset [27]”
BackgroundMethodInfluentialAdvances and challenges in log analysis
Oliner, A., Ganapathi, A., & Xu, W. (2012) · Communications of the ACM · doi:10.1145/2076450.2076466 · Cited by 290
Deep learning for anomaly detection in log data: A survey
Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100470 · Cited by 202
“Log data is generated in high volumes, with some systems producing millions [18] or even billions [19] of events daily.”
BackgroundAn Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2016) · 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2016.66 · Cited by 230
“The raw log messages, as shown in the figure, contain ten log messages extracted from HDFS log data on Amazon EC2 platform [2].”
BackgroundMethodInfluentialToward Large-Scale Vulnerability Discovery using Machine Learning
Grieco, G., Grinblat, G. L., Uzal, L., Rawat, S., Feist, J., & Mounier, L. (2016) · Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy · doi:10.1145/2857705.2857720 · Cited by 238
“Such approach is very similar to the ones already used to deal with traces in other works [32, 33].”
ResultLog-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
“Over the years, many data-driven methods have been proposed to automatically detect system anomalies by analyzing log data [6, 7, 12, 29, 47, 50, 54].”
Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network
Lu, S., Wei, X., Li, Y., & Wang, L. (2018) · 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech) · doi:10.1109/dasc/picom/datacom/cyberscitec.2018.00037 · Cited by 220
“For example, [14] leverages a decision tree to visualize the anomalies, and [16] uses a linear regression to compute the probability’s of abnormal tasks.”
BackgroundMethodResultInfluentialHitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3034647 · Cited by 228
“Existing log-based anomaly detection methods can be broadly classified into two categories: log event indexes based approaches (e.g., PCA [5], Invariant Mining [6], SVM [7], DeepLog [8]) and log template semantics-based approaches (e.g., LogAnomaly [9], LogRobust [10]).”
MethodInfluentialStructured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Cited by 243
DeepTraLog: trace-log combined microservice anomaly detection through graph-based deep learning
Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., & Zhang, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510180 · Cited by 207
“work on a single log event sequence such as the HDFS dataset [38].”
BackgroundBalanced multi-perspective checking of process conformance
Mannhardt, F., de Leoni, M., Reijers, H. A., & van der Aalst, W. M. P. (2016) · Computing · doi:10.1007/s00607-015-0441-1 · Cited by 222
“To our knowledge, the same limitation is also shared by approaches that use alternative languages to handle verification with data variables in processes (e.g. [4]), as well as by techniques to debug the execution of distributed systems (e.g. [19,24]).”
Towards Automated Log Parsing for Large-Scale Log Data Analysis
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2762673 · Cited by 230
“Furthermore, modern systems often integrate open-source software components written by hundreds of developers [3].”
BackgroundMethodInfluentialLearning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R., & Zhang, D. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.60 · Cited by 212
“For instance, logs have been widely analyzed for various tasks, such as anomaly detection [25], [42], problem diagnosis [33], [43], program verification [37], security monitoring [32], usage analysis [31], etc.”
MethodResultLeveraging existing instrumentation to automatically infer invariant-constrained models
Beschastnikh, I., Brun, Y., Schneider, S., Sloan, M., & Ernst, M. D. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025151 · Cited by 216
“Other prior work on mining systems logs focused on detecting dependencies [31], anomalies [22, 32, 46, 49], and performance debugging [40, 41].”
BackgroundCharacterizing logging practices in open-source software
Yuan, D., Park, S., & Zhou, Y. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227202 · Cited by 193
“Log analysis work: Many systems analyze the productionrun logs for post-mortem diagnosis [32], [34], [18], [17].”
BackgroundMethodIdentifying impactful service system problems via log analysis
He, S., Lin, Q., Lou, J.-G., Zhang, H., Lyu, M. R., & Zhang, D. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236083 · Cited by 201
“PCA projects the log sequence vectors into a subspace.”
BackgroundMethodInfluentialOnline System Problem Detection by Mining Patterns of Console Logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.19 · Cited by 203
Self-citationService-Generated Big Data and Big Data-as-a-Service: An Overview
Zheng, Z., Zhu, J., & Lyu, M. R. (2013) · 2013 IEEE International Congress on Big Data · doi:10.1109/bigdata.congress.2013.60 · Cited by 201
“Data mining technologies such as principal component analysis (PCA) and robust principal component analysis are also employed for identifying performance bottlenecks via modeling and mining the trace logs [5], [18].”
Method(2015) · doi:10.1145/2815400.2815415 · Cited by 202
Automated Dynamic Firmware Analysis at Scale: A Case Study on Embedded Web Interfaces
Costin, A., Zarras, A., & Francillon, A. (2016) · Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security · doi:10.1145/2897845.2897900 · Cited by 179
“ents: The failure analysis and determination in large-scale deployments [20] can be improved and automated in several ways. One way is to perform log pre-processing [86], log mining [65] and analysis [84]. This approach often uses clustering and machine learning techniques to classify an unknown execution of the syste…”
MethodAutomated IT system failure prediction: A deep learning approach
Zhang, K., Xu, J., Min, M. R., Jiang, G., Pelechrinis, K., & Zhang, H. (2016) · 2016 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2016.7840733 · Cited by 171
Logram: Efficient Log Parsing Using n-Gram Dictionaries
Dai, H., Li, H., Chen, C. S., Shang, W., & Chen, T.-H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3007554 · Cited by 170
Developing a predictive model of quality of experience for internet video
Balachandran, A., Sekar, V., Akella, A., Seshan, S., Stoica, I., & Zhang, H. (2013) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2534169.2486025 · Cited by 176
Experience Report: Deep Learning-based System Log Analysis for Anomaly DetectionPreprint
Chen, Z., Liu, J., Gu, W., Su, Y., & Lyu, M. R.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05908 · Cited by 229
“For instance, HDFS logs employ block id to record the operations embeddings learned with word2vec algorithms, e.g., FastText [15]”
InfluentialInferring models of concurrent systems from logs of their behavior with CSight
Beschastnikh, I., Brun, Y., Ernst, M. D., & Krishnamurthy, A. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568246 · Cited by 181
“In addition to inferring models, concurrent system logs can be used to detect anomalies [35, 44, 57], identify performance bugs [52, 53], and mine temporal system properties [9, 17, 59].”
BackgroundMi, H., Wang, H., Zhou, Y., Lyu, M. R.-T., & Cai, H. (2013) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.21 · Cited by 176
“Xu et al. [27] combine console logs with source codes to construct performance features and conducts the PCA [20] to detect problems.”
BackgroundThe mystery machine: end-to-end performance analysis of large-scale internet services
Chow, M., Meisner, D., Flinn, J., Peek, D., & Wenisch, T. F. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685066 · Cited by 166
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Jiang, Z., Liu, J., Chen, Z., Li, Y., Huang, J., Huo, Y., He, P., Gu, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643733 · Cited by 222
“As such, log analysis plays an essential role in the maintenance of software systems.”
Diagnosing performance changes by comparing request flows
Sambasivan, R. R., Zheng, A. X., Rosa, M. D., Krevat, E., Whitman, S., Stroucken, M., Wang, W. Y., Xu, L., & Ganger, G. R. (2011) · doi:10.5555/1972457.1972463 · Cited by 165
Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs
Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., & Kao, O. (2020) · 2020 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm50108.2020.00148 · Cited by 176
“[14] proposed using the Principal Component Analysis (PCA) method, where they assume that there are different sessions in a log file that can be easily identified by a session-id attached to each log entry.”
BackgroundMethodLength Matters: Clustering System Log Messages using Length of WordsPreprint
Shima, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03213 · Cited by 174
“Xu, el al. proposed a method using source code knowledge to infer message templates in [11].”
BackgroundMethodSwissLog: Robust and Unified Deep Learning Based Log Anomaly Detection for Diverse Faults
Li, X., Chen, P., Jing, L., He, Z., & Yu, G. (2020) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre5003.2020.00018 · Cited by 167
“Prior works [3], [4], [6]–[8], [16] mostly focus on sequential log anomalies and recognize them by detecting abnormal log sequence order . b) Log time interval change: Another kind of fault is performance issues, whose example is shown in Fig.”
BackgroundMethodInfluentialCalifornia fault lines: understanding the causes and impact of network failures
Turner, D., Levchenko, K., Snoeren, A. C., & Savage, S. (2010) · Proceedings of the ACM SIGCOMM 2010 conference · doi:10.1145/1851182.1851220 · Cited by 164
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 189
Improving software diagnosability via log enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2011) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1950365.1950369 · Cited by 157
Chen, P., Qi, Y., Zheng, P., & Hou, D. (2014) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2014.6848128 · Cited by 154
Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection
Bertero, C., Roy, M., Sauvanaud, C., & Trédan, G. (2017) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2017.43 · Cited by 154
“In [22], the author proposes a new algorithm for the clustering of log events and implements a tool based on it named SLCT. Logfiles parsing is exploited in [24].”
A Lightweight Algorithm for Message Type Extraction in System Application Logs
Makanju, A., Zincir-Heywood, N., & Milios, E. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.138 · Cited by 152
Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Rodrigues, X., Zhang, Y., Jain, P. U., Stumm, M., & Stumm (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685068 · Cited by 151
Incremental Mining of System Log Format
Mizutani, M. (2013) · IEEE International Conference on Services Computing · doi:10.1109/scc.2013.73 · Cited by 151
“In some case, logs that have most important log type for troubleshooting are very rare and there is possibility that clustering approach drops them.”
BackgroundOnline aggregation and continuous query support in MapReduce
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Gerth, J. A., Talbot, J., Elmeleegy, K., & Sears, R. (2010) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1807167.1807295 · Cited by 149
“MapReduce is often used to analyze streams of constantly-arriving data, such as URL access logs [4] and system console logs [11].”
MethodUniParser: A Unified Log Parser for Heterogeneous Log Data
Liu, Y., Zhang, X., He, S., Zhang, H., Li, L., Kang, Y., Xu, Y. D., Ma, M., Lin, Q., Dang, Y., et al. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511993 · Cited by 149
“, PCA [24], LSTM [6]) for further analysis.”
BackgroundMethodCarat: collaborative energy diagnosis for mobile devices
Oliner, A. J., Iyer, A. P., Stoica, I., Lagerspetz, E., & Tarkoma, S. (2013) · Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2517351.2517354 · Cited by 148
“Recent work shows how access to source code can facilitate tasks like log analysis [43] and distributed diagnosis [12].”
BackgroundCharacterizing Logging Practices in Open-Source Software
Yuan, D., Park, S., & Zhou, Y. (2012) · Cited by 148
(2017) · doi:10.1145/3132747.3132749 · Cited by 144
Be conservative: enhancing failure diagnosis with proactive logging
Yuan, D., Park, S., Huang, P., Liu, Y., Lee, M. M., Tang, X., Zhou, Y., & Savage, S. (2013) · Cited by 143
A survey of AIOps methods for failure management
Notaro, P., Cardoso, J., & Gerndt, M. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3483424 · Cited by 142
A Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139
“[223] propose an approach to detect and explain system problems by mining console logs.”
BackgroundRobust Anomaly Detection and Backdoor Attack Detection Via Differential PrivacyPreprint
Du, M., Jia, R., & Song, D. X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07116 · Cited by 159
“Over the past decade this log dataset has been extensively used for research in system log anomaly detection ([15, 16, 5]).”
MethodOmegaLog: High-Fidelity Attack Investigation via Transparent Multi-layer Log Analysis
Hassan, W. U., Noureddine, M. A., Datta, P., & Bates, A. (2020) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2020.24270 · Cited by 183
“Xu et al. [59] analyzed console logs to learn common patterns by using machine learning and to detect abnormal log patterns at runtime.”
BackgroundUnderstanding and detecting real-world performance bugs
Jin, G., Song, L., Shi, X., Scherpelz, J., & Lu, S. (2012) · ACM SIGPLAN Notices · doi:10.1145/2345156.2254075 · Cited by 136
Event Logs for the Analysis of Software Failures: A Rule-Based Approach
Cinque, M., Cotroneo, D., & Pecchia, A. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.67 · Cited by 134
“Despite a number of works proposing log filtering and coalescence algorithms [15], [16], [17], [9], [4], [18], [10], the real problem with failure analysis is the scarce ability of current logs at reporting the right set of error events, i.e., the ones leading to failures.”
BackgroundLifelong Anomaly Detection Through Unlearning
Du, M., Chen, Z., Liu, C., Oak, R., & Song, D. X. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3363226 · Cited by 169
“Over the past decade this log dataset has been extensively used for research in system log parsing and anomaly detection [7, 8, 24, 44].”
BackgroundMethodSystem log clustering approaches for cyber security applications: A survey
Landauer, M., Skopik, F., Wurzenberger, M., & Rauber, A. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101739 · Cited by 131
Assisting developers of Big Data Analytics Applications when deploying on Hadoop clouds
Shang, W., Jiang, Z. M., Hemmati, H., Adams, B., Hassan, A. E., & Martin, P. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606586 · Cited by 130
“in [26], which uses the source code to understand the structure of the logs.”
BackgroundLog Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129
Spell: Online Streaming Parsing of Large Unstructured System Logs
Du, M., & Li, F. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2875442 · Cited by 125
“To alleviate the pain of diving into massive unstructured log data, in most prior work, the first and foremost step is to automatically parse the unstructured system logs to structured data [1], [2], [3], [5].”
BackgroundMethodInfluentialChen, B., & Jiang, Z. M. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9429-5 · Cited by 124
“2011), to detect abnormal runtime behavior for big data applications (Shang et al. 2013; Xu et al. 2009), to analyze the results of a load test (Jiang et al.”
MethodImproving Software Diagnosability via Log Enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2012) · ACM Transactions on Computer Systems · doi:10.1145/2110356.2110360 · Cited by 124
DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
Han, D., Wang, Z., Chen, W., Zhong, Y., Wang, S., Zhang, H., Yang, J., Shi, X., & Yin, X. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484589 · Cited by 121
“For DeepLog , we use HDFS dataset [57] which is the same dataset evaluated in its work.”
Which log level should developers choose for a new logging statement?
Li, H., Shang, W., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9456-2 · Cited by 140
“…that are generated in large-scale data centers often consist of voluminous messages and it is difficult for operators to detect noteworthy logs, Xu et al. (2009) propose a method to mine the rich source of log information in order to automatically detect system runtime problems within a short…”
BackgroundMethodLogLens: A Real-Time Log Analysis System
Debnath, B. K., Solaimani, M., Gulzar, M. A., Arora, N., Lumezanu, C., Xu, J., Zong, B., Zhang, H., Jiang, G., & Khan, L. R. (2018) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2018.00105 · Cited by 117
“Existing log analysis tools either use predefined regular expressions (RegEx) or source-code level information for log parsing [11, 16, 24].”
BackgroundMethodLightLog: A lightweight temporal convolutional network for log anomaly detection on the edge
Wang, Z., Tian, J., Fang, H., Chen, L., & Qin, J. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.108616 · Cited by 116
Concept Drift Adaptation Methods under the Deep Learning Framework: A Literature Review
Xiang, Q., Zi, L., Cong, X., & Wang, Y. (2023) · Applied Sciences · doi:10.3390/app13116515 · Cited by 112
“For example, the I-LSTM and ECNN algorithms are collected data, and CausalLSTM uses the HDFS dataset [107] and the Cybersecurity’s Intrusion Detection Evaluation dataset [108].”
MethodLogFiT: Log Anomaly Detection Using Fine-Tuned Language Models
Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3358730 · Cited by 111
“Datasets: The LogFiT model is trained and evaluated using three public datasets: HDFS [46], BGL [47] and Thunderbird [47], as used by baseline models [8], [9].”
MethodLogzip: Extracting Hidden Structures via Iterative Clustering for Log Compression
Liu, J., Zhu, J., He, S., He, P., Zheng, Z., & Lyu, M. R.-T. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00085 · Cited by 111
de Leoni, M., & van der Aalst, W. M. P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40176-3_10 · Cited by 111
“For this purpose, one could also leverage on existing techniques in the field of distributed systems for system debugging [14, 15].”
Backgroundlprof: a non-intrusive request flow profiler for distributed systems
Zhao, X., Zhang, Y., Lion, D., Ullah, M. F., Luo, Y., Yuan, D., & Stumm, M. (2014) · TSpace (University of Toronto) · doi:10.5555/2685048.2685099 · Cited by 111
Brain: Log Parsing With Bidirectional Parallel Tree
Yu, S., He, P., Chen, N., & Wu, Y. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3270566 · Cited by 110
“Existing log parsing approaches mainly include regular expression filters [5], frequent item mining [7], [8], heuristic [9], clustering [10] and neural network [11], [12].”
BackgroundMethodBERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model
Chen, S., & Liao, H. (2022) · Applied Artificial Intelligence · doi:10.1080/08839514.2022.2145642 · Cited by 109
“Xu et al. (2009) use PCA algorithm to detect large-scale system problems by converting logs into count vectors.”
MethodDEEPCASE: Semi-Supervised Contextual Analysis of Security Events
van Ede, T., Aghakhani, H., Spahn, N., Bortolameotti, R., Cova, M., Continella, A., van Steen, M. R., Peter, A., Kruegel, C., & Vigna, G. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833671 · Cited by 108
“We also evaluate DEEPCASE on the HDFS dataset [44] used in the evaluation of the related security log”
MethodExamining the stability of logging statements
Kabinna, S., Bezemer, C., Shang, W., Syer, M. D., & Hassan, A. E. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9518-0 · Cited by 104
“Logs support various development activities such as fixing bugs (Xu et al. 2009; Lou et al. 2010; Fu et al. 2009), analyzing load tests (Malik et al. 2013), monitoring performance (Yuan et al. 2011) and transferring knowledge (Shang et al. 2014a).”
BackgroundCloud Analytics for Capacity Planning and Instant VM Provisioning
Jiang, Y., Perng, C.-S., Li, T., & Chang, R. (2013) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2013.051913.120278 · Cited by 103
Characterizing the natural language descriptions in software logging statements
He, P., Chen, Z., He, S., & Lyu, M. R.-T. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238193 · Cited by 103
“Specifically, software logs have been used in various reliability enhancement tasks, including anomaly detection [18, 57], fault diagnosis [55, 63], program verification [17, 51], performance monitoring [27, 43], etc.”
MethodLogGPT: Log Anomaly Detection via GPT
Han, X., Yuan, S., & Trabelsi, M. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386543 · Cited by 102
“Traditional models, such as Principal Component Analysis (PCA) [7], Isolation forest [8], and one-class Support Vector Machines (OCSVM) [9] have been widely used.”
MethodCloudSeer: Workflow Monitoring of Cloud Infrastructures via Interleaved Logs
Yu, X., Joshi, P., Xu, J., Jin, G., Zhang, H., & Jiang, G. (2016) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2872362.2872407 · Cited by 100
“Existing work [21, 24, 29] on detecting problems using data mining and clustering on logs can only operate in an offline manner, since they require task or program executions to finish so that logs are available in their entirety beforehand.”
BackgroundMethodEnCore: exploiting system environment and correlation information for misconfiguration detection
Zhang, J., Renganarayana, L., Zhang, X., Ge, N., Bala, V., Xu, T., & Zhou, Y. (2014) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2541940.2541983 · Cited by 99
“Since our goal is to extract the configuration correlations, either with the executing environment or between different configuration entries, the straightforward idea is to apply data mining methods to learn the association rules, as proposed in [34] and shown to be effective in other works [45, 47].”
MethodStudying software logging using topic models
Li, H., Chen, T.-H. P., Shang, W., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9595-8 · Cited by 99
“Such information can be used for a variety of software quality assurance tasks, such as debugging and understanding system usage in production (Chen et al. 2016a, 2017a; Mariani and Pastore 2008; Oliner et al. 2012; Syer et al. 2013; Xu et al. 2009; Yuan et al. 2010).”
BackgroundRuntime Analysis of Whole-System Provenance
Pasquier, T., Han, X., Moyer, T., Bates, A., Hermant, O., Eyers, D. M., Bacon, J., & Seltzer, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243776 · Cited by 99
“Timely investigation of system intrusions remains a notoriously difficult challenge [66, 94, 96].”
BackgroundA Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives
Li, H., Shang, W., Adams, B., Sayagh, M., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2970422 · Cited by 97
“Anomaly detection tools [7, 13, 36] automatically analyze large amounts of log messages (that are hard for human beings to investigate manually) and alert anomalies that are indicated in the log messages.”
BackgroundZhang, T., Qiu, H., Castellano, G., Rifai, M., Chen, C. S., & Pianese, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3222417 · Cited by 130
“Unfortunately, due to the privacy concerns with personal and commercial data, there are still only a few public log datasets [70], [92], [122], [132], [160].”
BackgroundMethodAI for DevSecOps: A Landscape and Future Opportunities
Fu, M. C., Pasuksmit, J., & Tantithamthavorn, C. K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712190 · Cited by 109
LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection
Guo, H., Yang, J., Liu, J., Bai, J., Wang, B., Li, Z., Zheng, T., Zhang, B., Peng, J., & Tian, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27764 · Cited by 94
LogSed: Anomaly Diagnosis through Mining Time-Weighted Control Flow Graph in Logs
Jia, T., Yang, L., Chen, P., Li, Y., Meng, F., & Xu, J. (2017) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2017.64 · Cited by 91
“Other works ([5][6]) leverage a white-box approach where source code is available and a duplicated controllable environment is available.”
BackgroundLog-Based Anomaly Detection With Robust Feature Extraction and Online Learning
Han, S., Wu, Q., Zhang, H., Qin, B., Hu, J., Shi, X., Liu, L., & Yin, X. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3053371 · Cited by 90
“A. Evaluation Methodology 1) Data Set: Three public log datasets [23] are used in our evaluation: a) HDFS Log Data Set: It is generated through running Hadoop-based map-reduce jobs on 203 Amazon’s EC2 nodes [47], and labeled by Hadoop domain experts.”
MethodInfluential
Metadata
- Authors
- Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I.
- Affiliations
- University of California, Berkeley (US)
- Intel (United States) (US)
- Venue
- Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles
- Publisher
- ACM
- Published
- 2009-10-11
- Pages
- 117-132
- OpenAlex
- W2039157918
- Cited by
- 1,332
- References
- 44
- Citations by year
- 2026: 100 · 2025: 126 · 2024: 134 · 2023: 138 · 2022: 97 · 2021: 126 · 2020: 96 · 2019: 89 · 2018: 72 · 2017: 49 · 2016: 62 · 2015: 48 · 2014: 41 · 2013: 37 · 2012: 42
- FWCI
- 31.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGOPS Symposium on Operating Systems Principles (SOSP) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault detection, log parsing, composite features, source code analysis, information retrieval, decision tree, Hadoop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.