The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
Tools and Benchmarks for Automated Log Parsing
Open accessAbstract
Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While computationally lightweight, these methods struggle to adapt to evolving workloads and often suffer from high false-positive rates when applied to complex, real-world clusters.”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“pretraining contamination can silently inflate benchmark performance, and contamination checks are now part of mainstream LLM evaluation practice [147], [44], [139].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLM graders can help with structure and readability, but they are not a substitute for tool-grounded correctness and safety scoring.”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This frequencyratio-based heuristic has been widely adopted in prior work on log anomaly detection, particularly in template blacklisting and rule construction [4], [5].”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332
Drain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943
Self-citationWhat Supercomputers Say: A Study of Five System Logs
Oliner, A. J., & Stearley, J. (2007) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2007.103 · Cited by 695
Experience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R.-T. (2016) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2016.21 · Cited by 675
Self-citationUsing magpie for request extraction and workload modelling
Barham, P., Donnelly, A., Isaacs, R., & Mortier, R. (2004) · Operating Systems Design and Implementation · Cited by 571
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J., Wang, Y., & Jiang, L. (2009) · Proceedings · doi:10.1109/icdm.2009.60 · Cited by 555
Log clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J., Zhang, Y., & Chen, X. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889232 · Cited by 493
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · IP Operations and Management · doi:10.1109/ipom.2003.1251233 · Cited by 480
Spell: Streaming Parsing of System Event Logs
Du, M., & Li, F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0103 · Cited by 423
Clustering event logs using iterative partitioning
Makanju, A., Zincir-Heywood, N., & Milios, E. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557154 · Cited by 369
Advances and challenges in log analysis
Oliner, A. J., Ganapathi, A., & Xü, W. (2012) · Communications of the ACM · doi:10.1145/2076450.2076466 · Cited by 318
Mining invariants from console logs for system problem detection
Lou, J., Fu, Q., Yang, S., Xu, Y., & Li, J. (2010) · Cited by 301
LogCluster - A data clustering and pattern mining algorithm for event logs
Vaarandi, R., & Pihelgas, M. (2015) · Conference on Network and Service Management · doi:10.1109/cnsm.2015.7367331 · Cited by 294
An Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.-T. (2016) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2016.66 · Cited by 274
Self-citationLogSig: generating system events from raw textual logs
Tang, L., Li, T., & Perng, C.-S. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063690 · Cited by 244
Where do developers log? an empirical study on logging practices in industry
Fu, Q., Zhu, J., Hu, W., Lou, J., Ding, R. F., Lin, Q., Zhang, D., & Xie, T. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591175 · Cited by 235
Self-citationStructured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Cited by 234
Hamooni, H., Debnath, B. K., Xu, J., Zhang, H., Jiang, G., & Mueen, A. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983358 · Cited by 230
Towards Automated Log Parsing for Large-Scale Log Data Analysis
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2762673 · Cited by 230
Self-citationLearning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R.-T., & Zhang, D. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.60 · Cited by 229
Self-citationYuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., & Pasupathy, S. (2010) · ACM SIGARCH Computer Architecture News · doi:10.1145/1735970.1736038 · Cited by 212
Identifying impactful service system problems via log analysis
He, S., Lin, Q., Lou, J., Zhang, H., Lyu, M. R.-T., & Zhang, D. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236083 · Cited by 205
Self-citationCharacterizing logging practices in open-source software
Yuan, D., Park, S., & Zhou, Y. (2012) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2012.6227202 · Cited by 195
Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data
Oprea, A., Li, Z., Yen, T.-F., Chin, S., & Alrwais, S. A. (2015) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2015.14 · Cited by 185
The mystery machine: end-to-end performance analysis of large-scale internet services
Chow, M., Meisner, D., Flinn, J., Peek, D., & Wenisch, T. F. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685066 · Cited by 182
A search-based approach for accurate identification of log message formats
Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., & Sasnauskas, R. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196340 · Cited by 177
Mi, H., Wang, H., Zhou, Y., Lyu, M. R.-T., & Cai, H. (2013) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.21 · Cited by 176
Self-citationYuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., & Pasupathy, S. (2010) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1736020.1736038 · Cited by 176
Abstracting log lines to log event types for mining software system logs
Nagappan, M., & Vouk, M. A. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463281 · Cited by 171
Improving software diagnosability via log enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2011) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1950365.1950369 · Cited by 157
Characterizing Logging Practices in Open-Source Software
Yuan, D., Park, S., & Zhou, Y. (2012) · Cited by 151
Incremental Mining of System Log Format
Mizutani, M. (2013) · IEEE International Conference on Services Computing · doi:10.1109/scc.2013.73 · Cited by 145
Assisting developers of Big Data Analytics Applications when deploying on Hadoop clouds
Shang, W., Jiang, Z. M., Hemmati, H., Adams, B., Hassan, A. E., & Martin, P. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606586 · Cited by 125
Characterizing the natural language descriptions in software logging statements
He, P., Chen, Z., He, S., & Lyu, M. R.-T. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238193 · Cited by 103
Self-citationZhao, X., Rodrigues, K., Luo, Y., Stumm, M., Yuan, D., & Zhou, Y. (2017) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3132747.3132778 · Cited by 101
The unified logging infrastructure for data analytics at Twitter
Lee, G., Lin, J., Liu, C., Lorek, A., & Ryaboy, D. V. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367516 · Cited by 90
Length Matters: Clustering System Log Messages using Length of WordsPreprint
Shima, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03213 · Cited by 89
Log 2 : a cost-aware logging mechanism for performance diagnosis
Ding, R. F., Zhou, H., Lou, J., Zhang, H., Lin, Q., Fu, Q., Zhang, D., & Xie, T. (2015) · NOVA (University of Newcastle Australia) · Cited by 70
Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays
Nagappan, M., Wu, K. J., & Vouk, M. A. (2009) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.23 · Cited by 65
POD-Diagnosis: Error Diagnosis of Sporadic Operations on Cloud Applications
Xu, X., Zhu, L., Weber, I. M., Bass, L., & Sun, D. W. (2014) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2014.94 · Cited by 62
Navigating the Data Lake with DATAMARAN
Gao, Y., Huang, S., & Parameswaran, A. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3183746 · Cited by 51
System problem detection by mining console logs
Patterson, D. A., & Xu, W. (2010) · Cited by 37
Identifying Recurrent and Unknown Performance Issues
Lim, M.-H., Lou, J., Zhang, H., Fu, Q., Teoh, A. B. J., Lin, Q., Ding, R. F., & Zhang, D. (2014) · Proceedings · doi:10.1109/icdm.2014.96 · Cited by 37
Mining Historical Issue Repositories to Heal Large-Scale Online Service Systems
Ding, R. F., Fu, Q., Lou, J., Lin, Q., Zhang, D., & Xie, T. (2014) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2014.39 · Cited by 31
Towards a neural language model for signature extraction from forensic logs
Thaler, S., Menkonvski, V., & Petković, M. (2017) · International Symposium on Digital Forensic and Security (ISDFS) · doi:10.1109/isdfs.2017.7916497 · Cited by 25
Abstracting Execution Logs to Execution Events for Enterprise Applications
Jiang, Z. M., Hassan, A. E., Flora, P. K., & Hamann, G. (2008) · Cited by 18
Improving software diagnosability via log enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1961296.1950369
Research works printed but not linked: 12
- 0, New automated log parsing
- 0, Logz io
- 0, Overview of logs-based metrics
- 0, Loghub A collection of system log datasets for intelligent log analysis
- 0, Automated root cause analysis for spark application failures
- 0, Log Management Service in the Cloud-Loggly
- 0, Log parsing - automated easy to use and efficient
- 0, Splunk
- 0, ElK
- 0, Logentries
- 0, A beginner? guide to logstash grok
- 0, Automated parsing log types
Cited by
Sorted by citations
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“Log parsers could produce inconsistent results depend on the preprocessing step and the set of parameters [16], [15].”
MethodInfluentialLoghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302
Self-citationSemi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00130 · Cited by 276
Log-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
Metadata
- Authors
- Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R.
- Affiliations
- Huawei Technologies (China) (CN)
- Chinese University of Hong Kong (HK)
- Sun Yat-sen University (CN)
- ETH Zurich (CH)
- Southwest Minzu University (CN)
- Venue
- 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 121-130
- OpenAlex
- W2963999143
- Cited by
- 574
- References
- 49
- Citations by year
- 2026: 57 · 2025: 86 · 2024: 108 · 2023: 92 · 2022: 62 · 2021: 89 · 2020: 65 · 2019: 15
- FWCI
- 49.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61332010, 61502401, 61722214)
- Chinese University of Hong Kong (14210717)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- benchmarking, structured data extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
