Towards Automated Log Parsing for Large-Scale Log Data Analysis
The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index.
Abstract
Logs are widely used in system management for dependability assurance because they are often the only data available that record detailed system runtime behaviors in production. Because the size of logs is constantly increasing, developers (and operators) intend to automate their analysis by applying data mining methods, therefore structured input data (e.g., matrices) are required. This triggers a number of studies on log parsing that aims to transform free-text log messages into structured events. However, due to the lack of open-source implementations of these log parsers and benchmarks for performance comparison, developers are unlikely to be aware of the effectiveness of existing log parsers and their limitations when applying them into practice. They must often reimplement or redesign one, which is time-consuming and redundant. In this paper, we first present a characterization study of the current state of the art log parsers and evaluate their efficacy on five real-world datasets with over ten million log messages. We determine that, although the overall accuracy of these parsers is high, they are not robust across all datasets. When logs grow to a large scale (e.g., 200 million log messages), which is common in practice, these parsers are not efficient enough to handle such data on a single computer. To address the above limitations, we design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform. Comprehensive experiments have been conducted to evaluate POP on both synthetic and real-world datasets. The evaluation results demonstrate the capability of POP in terms of accuracy, efficiency, and effectiveness on subsequent log mining tasks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Everitt, B. J., Landau, S., & Leese, M. (1974) · Cited by 9,363
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Everitt, B. S., Landau, S., Leese, M., & Stahl, D. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470977811 · Cited by 2,907
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 10×
“Furthermore, modern systems often integrate open-source software components written by hundreds of developers [3].”
BackgroundMethodInfluentialMinimum Spanning Trees and Single Linkage Cluster Analysis
Gower, J. C., & Ross, G. J. S. (1969) · Applied Statistics · doi:10.2307/2346439 · Cited by 1,223 · cited 1×
“To address over-parsing and further improve parsing accuracy, in this step, POP employs hierarchical clustering [27] to cluster similar groups based on their log events.”
MethodDiagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039 · cited 1×
“However, the anomaly detection task evaluated is an important log mining task widely studied [37], [38], which is presented in a paper [3] enjoying more than 300 citations.”
BackgroundWhat Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620 · cited 2×
“Besides, we thank those who release log data [3], [33], [34], which greatly facilitates our research.”
MethodThe Complexity of Some Problems on Subsequences and Supersequences
Maier, D. (1978) · Journal of the ACM · doi:10.1145/322063.322075 · Cited by 616 · cited 1×
“The groups in the same cluster will be merged, and a new log event will be generated by calculating the Longest Common Subsequence (LCS) [28] of the original log events.”
Method
Research works printed but not linked: 4
- salfner, 2004, Comprehensive logfiles for autonomic systems, Proc 18th Int Parallel Distrib Process Symp
- krause, 1987, Taxicab Geometry
- gu, 2015, Proc 45th Annu IEEE/IFIP Int Conf Dependable Syst Netw, 57
- lang, 2013, Using SEC, LOGIN The USENIX Magazine, 38, 38
Cited by
Items 201–231 of 231
Peer Review #1 of "Log-based software monitoring: a systematic mapping study (v0.2)"
(2021) · PeerJ Computer Science · doi:10.7287/peerj-cs.489v0.2/reviews/1
2018 Index IEEE Transactions on Dependable and Secure Computing Vol. 15
(2019) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2018.2885264
A Network Infrastructure for Security Enhancement
Verma, S. (2022) · Soft Computing Research Society eBooks · doi:10.52458/978-81-955020-5-9-14
Context‐Aware Learning for Robust Anomaly Detection *
Wu, Y., Ge, J., & Li, T. (2022) · Wiley eBooks · doi:10.1002/9781119835905.ch7
Analysis Of the Behavior of Cyberattacks on Online Services Using the Cyber Threat Classification
Владиславович, И. С., & Дмитриевич, К. Д. (2022) · Journal of Systems Engineering and Information Technology (JOSEIT) · doi:10.29207/joseit.v1i2.4944
ReLog: A Novel Method for Log Recognition
Tian, B., Wang, Z., Chen, G., Zhang, Y., & Zhou, H. (2023) · IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE) · doi:10.1109/auteee60196.2023.10407537
“…by generating a hierarchy of patterns; Adetokunbo A.O [9] proposed a method for partitioning logs based on some pre-existing assumptions; He P [10] utilized the log length feature and lexical position feature to plan log partitions and cluster the resulting partitions using Manhattan…”
MethodNon-Deterministic and Semi-Supervised Log-based Anomaly Detection
Kumar, S., Murgai, V., Bashir, Z., & Singh, S. (2023) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps58843.2023.10464781
“…sequential models cannot learn patterns from this data, so log parsers represent each log entry as a set of log key events, which refer to the text string printed during the execution of source code by a system, and parameter values, which refer to the numerical values in a log message [7] [10].”
BackgroundMethodAnomaly Detection from System Logs based on Deep Learning Techniques
Lv, Y., & Min, S. (2023) · International Conference on Electrical, Mechanical and Computer Engineering (ICEMCE) · doi:10.1109/icemce60359.2023.10490680
“In [5,6], log sequences are converted into log vectors, and support vector machines and logistic regression are used for anomaly detection.”
MethodAnomaly Detection for Log using AutoM and Auto-Sklearn Classification Algorithm
Karthik, G. M., Gupta, A., Gupta, S. R., Jha, A., Sivasangari, A., & Mishra, B. P. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724030
Mapping Cyberattack Patterns and Detection: An Azure Sentinel Approach
Odigbo, C. S., Wimmer, H., & Kim, J. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-82610-8_14
Learning the fine-grained code representation for log-level prediction
Zhao, Z., Fan, G., Jing, L., Zhu, M., Zhang, H., & Su, H. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00064-9
A Network Scanning Organization Discovery Method Based on Graph Convolutional Neural Network
Xue, P., Dong, L., Wang, C., Huang, C., & Wang, J. (2025) · Information · doi:10.3390/info16100899
United We Log, Divided We Identify: A Decentralized Approach for Automated Log Analysis
Rabieinejad, E., Dehghantanha, A., Zarrinkalam, F., & Schwartzentruber, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01823-6_5 · Cited by 1
Log Anomaly Detection Based on Autonomous Attention Mechanism
Liu, J., & Fang, Z. (2025) · International Conference on Internet of Things, Artificial Intelligence and Mechanical Automation (IoTAIMA) · doi:10.1109/iotaima66468.2025.11212701
“He S et al.[10] summarized and compared several advanced machine learning methods used for log anomaly detection in recent years.”
ClusterRCA: An End-to-End Approach for Network Fault Localization and Classification for HPC System
Sun, Y., Pan, X., Xiong, X., Tao, L., Wang, J., Zhang, S., Yuan, Y., Li, Y., & Jian, K. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00043 · Cited by 3
LPB-Gen: Systematic Large Log-Parsing Benchmarks Generation
Dai, H., Yao, K., Li, F., You, J., Shen, Q., & Shang, W. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3787494
“In this section, we conduct comprehensive user studies to evaluate the correctness of the log benchmark data that is produced by LPB-Gen, as well as the efficiency and reproducibility of LPB-Gen.”
AI-Powered Log Analysis with BERT for Automated Event Detection and Categorization
Shine Krishon, T., Sathishwaran, S., & Raghavi, S. (2025) · International Conference on Artificial Intelligence and Knowledge Discovery in Concurrent Engineering (ICECONF) · doi:10.1109/iceconf65644.2025.11379598
“LogBERT [5] represents the first application of BERT to log anomaly detection, demonstrating the potential of trans-former models for understanding log semantics.”
LogFree: A Low-Resource-Oriented Model for Log Anomaly Detection
Li, Y., Wei, Y., Gu, T., Yu, M., & Lin, Y. (2025) · International Conference on Blockchain Technology and Information Security (ICBCTIS) · doi:10.1109/icbctis66509.2025.11387440
“The Support Vector Machine-based method (SVM) [7] exhibits superior capability in handling non-linear log patterns.”
Towards Automation in Log Parsing: Auto-Prompt Optimization with Natural Language Gradients
Chen, J., Hong, Y., Liu, J., & Bu, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_25
LogSNN: Spiking Neural Networks for System Log Anomaly DetectionPreprint
Chen, S., Liao, H., Liang, Y., Zhou, H., & Min, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6343052
Robust and efficient log anomaly detection: A hybrid ID-semantic approach for evolving systems
Alzahrani, M. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108105 · Cited by 1
Automated Log-Based Anomaly Detection within Cloud Computing Infrastructures
Vervaet, A. (2023) · theses.fr (ABES) · doi:10.70675/ed5ee972zb50cz4f6bz8bcazc53a6f3b58f1 · Cited by 4
LogNER: Enhancing log semantics with LLM-driven entity recognition
Zhao, C., Xiang, J., Zhao, L., Yu, A., Cai, L., & Meng, D. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112892
ParSimMerge: A Scalable Log Parser based on Parallelized Similarity-based Merge Optimization
Zhao, Y. (2026) · International Conference on Big Data, Information and Computer Network (BDICN) · doi:10.1145/3801228.3801287
“An example is POP [17], which proposes a Spark-based parallel parsing strategy.”
LogRAIL: A Retrieval-Augmented LLM Reverification Layer for Log Anomaly Detection
WONGWANG, C., Park, D., Kim, M., Cho, S., 이성훈, Park, J., & Park, H. (2026) · IEEE Access · doi:10.1109/access.2026.3688834
“Several factors complicate cross-study comparisons, including datasets biased toward specific services or software versions, inconsistent criteria for normal and anomalous behavior, mismatches in experimental settings, and reporting practices [11].”
Menda, J. R. (2021) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst523105136
LogCraft: Pattern-Guided Iterative Clustering for Efficient Unsupervised LLM-Driven Log Parsing
Hao, Y., & Tian, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2759-4_13
An Empirical Study on the Prevalence of Rare Logs and its Impact of Log Parsing
Hodaeian, S. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3804437
“Log parsing is a critical step that transforms unstructured log data into structured formats as shown in Figure 1, enabling automated log analysis for tasks such as debugging [1, 2], monitoring [3, 13], root cause analysis [9, 11, 15], and anomaly detection [3, 4, 10, 12].”
ABRLog: a multi-device android bugreport log dataset for intelligent analysis
Filho, R. M., Bessa, J. A., & de Freitas, R. (2026) · Data in Brief · doi:10.1016/j.dib.2026.113230
“Their analysis enables developers, administrators, and researchers to identify performance issues, usage patterns, and potential system failures [ 2 , 3 ].”
Kozaeva, N., & Bade, K. (2026) · Journal of Rail Transport Planning & Management · doi:10.1016/j.jrtpm.2026.100611
Data-Driven Release Governance for Embedded Software Systems in Consumer Devices
Ravindran, R. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.5548
Metadata
- Authors
- He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.
- Affiliations
- Chinese University of Hong Kong, Shenzhen (CN)
- Huawei Technologies (China) (CN)
- Venue
- IEEE Transactions on Dependable and Secure Computing
- Published
- 2018-11-01
- Volume
- 15
- Issue
- 6
- Pages
- 931-944
- ISSN
- 1545-5971 (ISSN-L)
- 1941-0018
- 2160-9209
- OpenAlex
- W2762028850
- Cited by
- 230
- References
- 42
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 28 · 2023: 40 · 2022: 23 · 2021: 45 · 2020: 30 · 2019: 18 · 2018: 8
- FWCI
- 10.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Dependable and Secure Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61332010)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14205214)
- 2015 Microsoft Research Asia Collaborative Research Program (FY16-RES-THEME-005)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log parsing, log mining, Apache Spark, dependability assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.