The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
Tools and Benchmarks for Automated Log Parsing
Open accessAbstract
Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While computationally lightweight, these methods struggle to adapt to evolving workloads and often suffer from high false-positive rates when applied to complex, real-world clusters.”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“pretraining contamination can silently inflate benchmark performance, and contamination checks are now part of mainstream LLM evaluation practice [147], [44], [139].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLM graders can help with structure and readability, but they are not a substitute for tool-grounded correctness and safety scoring.”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This frequencyratio-based heuristic has been widely adopted in prior work on log anomaly detection, particularly in template blacklisting and rule construction [4], [5].”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332
Drain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943
Self-citationWhat Supercomputers Say: A Study of Five System Logs
Oliner, A. J., & Stearley, J. (2007) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2007.103 · Cited by 695
Experience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R.-T. (2016) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2016.21 · Cited by 675
Self-citationUsing magpie for request extraction and workload modelling
Barham, P., Donnelly, A., Isaacs, R., & Mortier, R. (2004) · Operating Systems Design and Implementation · Cited by 571
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J., Wang, Y., & Jiang, L. (2009) · Proceedings · doi:10.1109/icdm.2009.60 · Cited by 555
Log clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J., Zhang, Y., & Chen, X. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889232 · Cited by 493
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · IP Operations and Management · doi:10.1109/ipom.2003.1251233 · Cited by 480
Spell: Streaming Parsing of System Event Logs
Du, M., & Li, F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0103 · Cited by 423
Research works printed but not linked: 12
- 0, New automated log parsing
- 0, Logz io
- 0, Overview of logs-based metrics
- 0, Loghub A collection of system log datasets for intelligent log analysis
- 0, Automated root cause analysis for spark application failures
- 0, Log Management Service in the Cloud-Loggly
- 0, Log parsing - automated easy to use and efficient
- 0, Splunk
- 0, ElK
- 0, Logentries
- 0, A beginner? guide to logstash grok
- 0, Automated parsing log types
Cited by
Sorted by citations
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“Log parsers could produce inconsistent results depend on the preprocessing step and the set of parameters [16], [15].”
MethodInfluentialLoghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302
Self-citationSemi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00130 · Cited by 276
Log-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
Metadata
- Authors
- Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R.
- Affiliations
- Huawei Technologies (China) (CN)
- Chinese University of Hong Kong (HK)
- Sun Yat-sen University (CN)
- ETH Zurich (CH)
- Southwest Minzu University (CN)
- Venue
- 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 121-130
- OpenAlex
- W2963999143
- Cited by
- 574
- References
- 49
- Citations by year
- 2026: 57 · 2025: 86 · 2024: 108 · 2023: 92 · 2022: 62 · 2021: 89 · 2020: 65 · 2019: 15
- FWCI
- 49.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61332010, 61502401, 61722214)
- Chinese University of Hong Kong (14210717)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- benchmarking, structured data extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
