Tools and benchmarks for automated log parsing
Open accessAbstract
Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional rule-based approaches struggle to adapt to evolving workloads and often suffer from high false-positive rates on complex, real-world clusters.
“While computationally lightweight, these methods struggle to adapt to evolving workloads and often suffer from high false-positive rates when applied to complex, real-world clusters.”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
The Introduction (p. 2) says handcrafted log-parsing rules require regular revision as logging code changes, and Industrial Deployment (p. 8) describes rule maintenance becoming difficult as System X evolves. This supports the adaptation concern, but our copy does not report high false-positive rates for rule-based methods on complex, real-world clusters; it discusses log parsing rather than that claimed outcome. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPretraining contamination can silently inflate benchmark performance.
“pretraining contamination can silently inflate benchmark performance, and contamination checks are now part of mainstream LLM evaluation practice [147], [44], [139].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
The Evaluation section (p. 5) benchmarks automated log parsers, and Robustness of Log Parsers (p. 7) discusses accuracy changes with log-data volume. Neither those passages nor searches for pretraining contamination and training-data leakage address inflated LLM benchmark performance. (LLM, conf=0.76)
Experimental — accuracy not yet measuredLLM graders are not a substitute for tool-grounded correctness and safety scoring.
“LLM graders can help with structure and readability, but they are not a substitute for tool-grounded correctness and safety scoring.”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [147]
The Evaluation section (p. 5) assesses log parsers for accuracy, robustness, and efficiency; the Experimental Setup (p. 6) defines correctness against ground-truth log-message groupings. Searches for LLM graders, readability, tool-grounded scoring, and safety found no passage addressing whether LLM graders can substitute for correctness and safety scoring. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe frequency-ratio-based heuristic has been widely adopted in prior work on log anomaly detection, particularly in template blacklisting and rule construction.
“This frequencyratio-based heuristic has been widely adopted in prior work on log anomaly detection, particularly in template blacklisting and rule construction [4], [5].”
From LINEADAPTER: Parameter-Efficient Fine-Tuning for Log Anomaly Detection and Root Cause Analysis (2025) · cites this paper as [5]
The Techniques of Log Parsers section (p. 4) describes frequency-based methods for extracting log templates, while Related Work (p. 9) lists anomaly detection as an application of log analysis. Neither passage addresses a frequency-ratio heuristic for anomaly detection or its adoption in template blacklisting and rule construction; searches for those terms found no supporting passage. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835 · see citing works
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · see citing works
Drain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · see citing works
Self-citationWhat Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · Dependable Systems and Networks · doi:10.1109/dsn.2007.103 · Cited by 721 · see citing works
Experience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R.-T. (2016) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2016.21 · Cited by 622 · see citing works
Self-citationUsing magpie for request extraction and workload modelling
Barham, P., Donnelly, A., Isaacs, R., & Mortier, R. (2004) · Operating Systems Design and Implementation · Cited by 571 · see citing works
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J., Wang, Y., & Jiang, L. (2009) · Proceedings · doi:10.1109/icdm.2009.60 · Cited by 555 · see citing works
Log clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J., Zhang, Y., & Chen, X. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889232 · Cited by 493 · see citing works
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · doi:10.1109/ipom.2003.1251233 · Cited by 461 · see citing works
Spell: Streaming Parsing of System Event Logs
Du, M., & Li, F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0103 · Cited by 423 · see citing works
Clustering event logs using iterative partitioning
Makanju, A., Zincir-Heywood, N., & Milios, E. (2009) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557154 · Cited by 346 · see citing works
Mining invariants from console logs for system problem detection
Lou, J., Fu, Q., Yang, S., Xu, Y., & Li, J. (2010) · Cited by 301 · see citing works
Advances and challenges in log analysis
Oliner, A. J., Ganapathi, A., & Xü, W. (2012) · Communications of the ACM · doi:10.1145/2076450.2076466 · Cited by 294 · see citing works
LogCluster - A data clustering and pattern mining algorithm for event logs
Vaarandi, R., & Pihelgas, M. (2015) · Conference on Network and Service Management · doi:10.1109/cnsm.2015.7367331 · Cited by 294 · see citing works
An Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.-T. (2016) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2016.66 · Cited by 245 · see citing works
Self-citationLogSig: generating system events from raw textual logs
Tang, L., Li, T., & Perng, C.-S. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063690 · Cited by 244 · see citing works
Where do developers log? an empirical study on logging practices in industry
Fu, Q., Zhu, J., Hu, W., Lou, J., Ding, R. F., Lin, Q., Zhang, D., & Xie, T. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591175 · Cited by 235 · see citing works
Self-citationStructured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Cited by 234 · see citing works
Hamooni, H., Debnath, B. K., Xu, J., Zhang, H., Jiang, G., & Mueen, A. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983358 · Cited by 230 · see citing works
Towards Automated Log Parsing for Large-Scale Log Data Analysis
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2762673 · Cited by 230 · see citing works
Self-citationLearning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R.-T., & Zhang, D. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.60 · Cited by 217 · see citing works
Self-citationYuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., & Pasupathy, S. (2010) · ACM SIGARCH Computer Architecture News · doi:10.1145/1735970.1736038 · Cited by 212 · see citing works
Identifying impactful service system problems via log analysis
He, S., Lin, Q., Lou, J., Zhang, H., Lyu, M. R.-T., & Zhang, D. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236083 · Cited by 205 · see citing works
Self-citationCharacterizing logging practices in open-source software
Yuan, D., Park, S., & Zhou, Y. (2012) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2012.6227202 · Cited by 195 · see citing works
Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data
Oprea, A., Li, Z., Yen, T.-F., Chin, S., & Alrwais, S. A. (2015) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2015.14 · Cited by 185 · see citing works
The mystery machine: end-to-end performance analysis of large-scale internet services
Chow, M., Meisner, D., Flinn, J., Peek, D., & Wenisch, T. F. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685066 · Cited by 182 · see citing works
Mi, H., Wang, H., Zhou, Y., Lyu, M. R.-T., & Cai, H. (2013) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.21 · Cited by 176 · see citing works
Self-citationYuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., & Pasupathy, S. (2010) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1736020.1736038 · Cited by 176 · see citing works
Abstracting log lines to log event types for mining software system logs
Nagappan, M., & Vouk, M. A. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463281 · Cited by 171 · see citing works
A search-based approach for accurate identification of log message formats
Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., & Sasnauskas, R. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196340 · Cited by 168 · see citing works
Improving software diagnosability via log enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2011) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1950365.1950369 · Cited by 157 · see citing works
Characterizing Logging Practices in Open-Source Software
Yuan, D., Park, S., & Zhou, Y. (2012) · Cited by 151 · see citing works
Incremental Mining of System Log Format
Mizutani, M. (2013) · IEEE International Conference on Services Computing · doi:10.1109/scc.2013.73 · Cited by 145 · see citing works
Assisting developers of Big Data Analytics Applications when deploying on Hadoop clouds
Shang, W., Jiang, Z. M., Hemmati, H., Adams, B., Hassan, A. E., & Martin, P. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606586 · Cited by 125 · see citing works
Zhao, X., Rodrigues, K., Luo, Y., Stumm, M., Yuan, D., & Zhou, Y. (2017) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3132747.3132778 · Cited by 101 · see citing works
Characterizing the natural language descriptions in software logging statements
He, P., Chen, Z., He, S., & Lyu, M. R.-T. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238193 · Cited by 95 · see citing works
Self-citationThe unified logging infrastructure for data analytics at Twitter
Lee, G., Lin, J., Liu, C., Lorek, A., & Ryaboy, D. V. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367516 · Cited by 90 · see citing works
Length Matters: Clustering System Log Messages using Length of WordsPreprint
Shima, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03213 · Cited by 89 · see citing works
Log 2 : a cost-aware logging mechanism for performance diagnosis
Ding, R. F., Zhou, H., Lou, J., Zhang, H., Lin, Q., Fu, Q., Zhang, D., & Xie, T. (2015) · NOVA (University of Newcastle Australia) · Cited by 70 · see citing works
Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays
Nagappan, M., Wu, K. J., & Vouk, M. A. (2009) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.23 · Cited by 65 · see citing works
POD-Diagnosis: Error Diagnosis of Sporadic Operations on Cloud Applications
Xu, X., Zhu, L., Weber, I. M., Bass, L., & Sun, D. W. (2014) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2014.94 · Cited by 62 · see citing works
Navigating the Data Lake with DATAMARAN
Gao, Y., Huang, S., & Parameswaran, A. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3183746 · Cited by 51 · see citing works
System problem detection by mining console logs
Patterson, D. A., & Xu, W. (2010) · Cited by 38 · see citing works
Identifying Recurrent and Unknown Performance Issues
Lim, M.-H., Lou, J., Zhang, H., Fu, Q., Teoh, A. B. J., Lin, Q., Ding, R. F., & Zhang, D. (2014) · Proceedings · doi:10.1109/icdm.2014.96 · Cited by 37 · see citing works
Mining Historical Issue Repositories to Heal Large-Scale Online Service Systems
Ding, R. F., Fu, Q., Lou, J., Lin, Q., Zhang, D., & Xie, T. (2014) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2014.39 · Cited by 31 · see citing works
Towards a neural language model for signature extraction from forensic logs
Thaler, S., Menkonvski, V., & Petković, M. (2017) · International Symposium on Digital Forensic and Security (ISDFS) · doi:10.1109/isdfs.2017.7916497 · Cited by 25 · see citing works
Abstracting Execution Logs to Execution Events for Enterprise Applications
Jiang, Z. M., Hassan, A. E., Flora, P. K., & Hamann, G. (2008) · Cited by 18 · see citing works
Improving software diagnosability via log enhancement
Yuan, D., Zheng, J., Park, S., Zhou, Y., & Savage, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1961296.1950369
Flagged first, then by citations
1 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [5][5] D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy, 'Sherlog: error diagnosis by connecting clues from run-time logs,' in ASPLOS , 2010, pp. 143-154. -
Cited by
Sorted by citations
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761 · see citing works
Loghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302 · see citing works
Self-citationLog-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 289 · see citing works
“Log parsers could produce inconsistent results depend on the preprocessing step and the set of parameters [16], [15].”
MethodInfluentialSemi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00130 · Cited by 276 · see citing works
Log-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257 · see citing works
HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3034647 · Cited by 234 · see citing works
Deep learning for anomaly detection in log data: A survey
Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100470 · Cited by 244 · see citing works
Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs
Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., & Kao, O. (2020) · 2020 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm50108.2020.00148 · Cited by 176 · see citing works
Logram: Efficient Log Parsing Using n-Gram Dictionaries
Dai, H., Li, H., Chen, C. S., Shang, W., & Chen, T.-H. P. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3007554 · Cited by 175 · see citing works
SwissLog: Robust and Unified Deep Learning Based Log Anomaly Detection for Diverse Faults
Li, X., Chen, P., Jing, L., He, Z., & Yu, G. (2020) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre5003.2020.00018 · Cited by 158 · see citing works
UniParser: A Unified Log Parser for Heterogeneous Log Data
Liu, Y., Zhang, X., He, S., Zhang, H., Li, L., Kang, Y., Xu, Y. D., Ma, M., Lin, Q., Dang, Y., et al. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511993 · Cited by 134 · see citing works
Self-citationLog Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129 · see citing works
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Jiang, Z., Liu, J., Chen, Z., Li, Y., Huang, J., Huo, Y., He, P., Gu, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643733 · Cited by 148 · see citing works
Self-citationUnsupervised log message anomaly detection
Farzad, A., & Gulliver, T. A. (2020) · ICT Express · doi:10.1016/j.icte.2020.06.003 · Cited by 125 · see citing works
LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning
Chen, R., Zhang, S., Li, D., Zhang, Y., Guo, F., Meng, W., Pei, D., Zhang, Y., Chen, X., & Liu, Y. (2020) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre5003.2020.00013 · Cited by 112 · see citing works
System log clustering approaches for cyber security applications: A survey
Landauer, M., Skopik, F., Wurzenberger, M., & Rauber, A. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101739 · Cited by 118 · see citing works
LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things
Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., & Tolba, A. K. R. (2020) · Sensors · doi:10.3390/s20092451 · Cited by 109 · see citing works
Self-citationLogzip: Extracting Hidden Structures via Iterative Clustering for Log Compression
Liu, J., Zhu, J., He, S., He, P., Zheng, Z., & Lyu, M. R.-T. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00085 · Cited by 102 · see citing works
Self-citationBrain: Log Parsing With Bidirectional Parallel Tree
Yu, S., He, P., Chen, N., & Wu, Y. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3270566 · Cited by 102 · see citing works
Self-citationPrediction and Detection of FDIA and DDoS Attacks in 5G Enabled IoT
Moudoud, H., Khoukhi, L., & Cherkaoui, S. (2020) · IEEE Network · doi:10.1109/mnet.011.2000449 · Cited by 100 · see citing works
Zuo, Y., Wu, Y., Min, G., Huang, C., & Pei, K. (2020) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2020.2966615 · Cited by 94 · see citing works
Identifying bad software changes via multimodal anomaly detection for online service systems
Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Hong-yu, X., Zhang, W., Sui, K., & Pei, D. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468543 · Cited by 97 · see citing works
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Ma, Z.-Y., Chen, A.-R., Kim, D. J., Chen, T., & Wang, S.-W. (2024) · International Conference on Software Engineering · doi:10.1145/3597503.3639150 · Cited by 151 · see citing works
Guidelines for assessing the accuracy of log message template identification techniques
Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510101 · Cited by 91 · see citing works
BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model
Chen, S., & Liao, H. (2022) · Applied Artificial Intelligence · doi:10.1080/08839514.2022.2145642 · Cited by 94 · see citing works
A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?
Jiang, Z., Liu, J., Huang, J., Li, Y., Huo, Y., Gu, J., Chen, Z., Zhu, J., & Lyu, M. R.-T. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652123 · Cited by 99 · see citing works
Self-citationLog-Based Anomaly Detection With Robust Feature Extraction and Online Learning
Han, S., Wu, Q., Zhang, H., Qin, B., Hu, J., Shi, X., Liu, L., & Yin, X. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3053371 · Cited by 88 · see citing works
Root-Cause Metric Location for Microservice Systems via Log Anomaly Detection
Wang, L., Zhao, N., Chen, J., Li, P., Zhang, W., & Sui, K. (2020) · 2020 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws49710.2020.00026 · Cited by 84 · see citing works
SwissLog: Robust Anomaly Detection and Localization for Interleaved Unstructured Logs
Li, X., Chen, P., Jing, L., He, Z., & Yu, G. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3162857 · Cited by 79 · see citing works
Zhang, T., Qiu, H., Castellano, G., Rifai, M., Chen, C. S., & Pianese, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3222417 · Cited by 82 · see citing works
DivLog: Log Parsing with Prompt Enhanced In-Context Learning
Xu, J., Yang, R., Huo, Y., Zhang, C., & He, P. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639155 · Cited by 96 · see citing works
Self-citationAnomaly Detection in the Open World: Normality Shift Detection, Explanation, and Adaptation
Han, D., Wang, Z., Chen, W., Wang, K., Yu, R., Wang, S., Zhang, H., Wang, Z., Jin, M., Yang, J., et al. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.24830 · Cited by 77 · see citing works
An empirical investigation of practical log anomaly detection for online service systems
Zhao, N., Wang, H., Li, Z., Peng, X., Wang, G., Pan, Z., Wu, Y., Feng, Z., Wen, X., Zhang, W., et al. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3473933 · Cited by 77 · see citing works
Experience Report: Deep Learning-based System Log Analysis for Anomaly DetectionPreprint
Chen, Z., Liu, J., Gu, W., Su, Y., & Lyu, M. R.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05908 · Cited by 134 · see citing works
Self-citationNedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., & Kao, O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67667-4_8 · Cited by 78 · see citing works
A systematic literature review on automated log abstraction techniques
El-Masri, D., Petrillo, F. D. S., Guéhéneuc, Y., Hamou‐Lhadj, A., & Bouziane, A. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106276 · Cited by 74 · see citing works
Heterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention
Lee, C. Q. E., Yang, T., Chen, Z., Su, Y., Yang, Y., & Lyu, M. R.-T. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00148 · Cited by 75 · see citing works
Self-citationAnomaly Detection in Operating System Logs with Deep Learning-Based Sentiment Analysis
Studiawan, H., Sohel, F. A., & Payne, C. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3037903 · Cited by 69 · see citing works
PLELog: Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion52605.2021.00106 · Cited by 72 · see citing works
A Semantic-aware Representation Framework for Online Log Analysis
Meng, W., Liu, Y., Huang, Y., Zhang, S., Zaiter, F., Chen, B. J., & Pei, D. (2020) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn49398.2020.9209707 · Cited by 68 · see citing works
Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection
Yu, B., Yao, J., Fu, Q., Zhong, Z., Xie, H., Wu, Y., Ma, Y., & He, P. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623308 · Cited by 70 · see citing works
Self-citationLogEncoder: Log-Based Contrastive Representation Learning for Anomaly Detection
Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., Li, H., Zhu, D., & Qian, D. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3239522 · Cited by 67 · see citing works
A Survey on Log Anomaly Detection using Deep Learning
Yadav, R. B., Kumar, P., & Dhavale, S. (2020) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito48877.2020.9197818 · Cited by 64 · see citing works
LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models
Zhong, A., Mo, D., Liu, G., Liu, J., Lu, Q., Zhou, Q., Wu, J., Li, Q., & Wen, Q. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671810 · Cited by 66 · see citing works
Self-citationMulti-source Distributed System Data for AI-Powered Analytics
Nedelkoski, S., Bogatinovski, J., Mandapati, A. K., Becker, S., Cardoso, J., & Kao, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-44769-4_13 · Cited by 60 · see citing works
DeepSyslog: Deep Anomaly Detection on Syslog Using Sentence Embedding and Metadata
Zhou, J., Qian, Y., Zou, Q., Liu, P., & Xiang, J. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3201379 · Cited by 61 · see citing works
Onion: identifying incident-indicating logs for cloud systems
Zhang, X., Xu, Y. D., Qin, S., He, S., Qiao, B., Li, Z., Zhang, H., Li, X., Dang, Y., Lin, Q., et al. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473919 · Cited by 59 · see citing works
Self-citationLogClass: Anomalous Log Identification and Classification With Partial Labels
Meng, W., Liu, Y., Zhang, S., Zaiter, F., Zhang, Y., Huang, Y., Yu, Z., Zhang, Y., Song, L., Zhang, M., et al. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3055425 · Cited by 56 · see citing works
ROSE: Robust Searchable Encryption With Forward and Backward Security
Xu, P., Susilo, W., Wang, W., Chen, T., Wu, Q., Liang, K., & Jin, H. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3155977 · Cited by 55 · see citing works
LogGD: Detecting Anomalies from System Logs with Graph Neural Networks
Xie, Y., Zhang, H., & Babar, M. A. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00039 · Cited by 61 · see citing works
Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies
Liu, Y., Tao, S., Meng, W., Wang, J., Ma, W., Chen, Y., Zhao, Y., Yang, H., & Jiang, Y. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644408 · Cited by 68 · see citing works
An empirical study of log analysis at Microsoft
He, S., Zhang, X., He, P., Xu, Y. D., Li, L., Kang, Y., Ma, M., Wei, Y., Dang, Y., Rajmohan, S., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558963 · Cited by 53 · see citing works
Self-citationLog Parsing: How Far Can ChatGPT Go?
Le, V.-H., & Zhang, H. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00206 · Cited by 68 · see citing works
Automatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 65 · see citing works
Self-citationXia, B., Bai, Y., Yin, J., Li, Y., & Xu, J. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-10026-3 · Cited by 54 · see citing works
Did We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction
Li, Z., Luo, C., Chen, T.-H. P., Shang, W., He, S., Lin, Q., & Zhang, D. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00078 · Cited by 56 · see citing works
Self-citationZhang, C., Wang, X., Zhang, H., Zhang, H., & Han, P. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3125967 · Cited by 50 · see citing works
Impact of log parsing on deep learning-based anomaly detection
Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10533-w · Cited by 49 · see citing works
SPINE: a scalable log parser with feedback guidance
Wang, X., Zhang, X., Li, L., He, S., Zhang, H., Liu, Y., Zheng, L., Kang, Y., Lin, Q., Dang, Y., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549176 · Cited by 49 · see citing works
Self-citationSelf-Supervised Anomaly Detection from Distributed Traces
Bogatinovski, J., Nedelkoski, S., Cardoso, J., & Kao, O. (2020) · 2020 IEEE/ACM 13th International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc48980.2020.00054 · Cited by 50 · see citing works
MLog: Mogrifier LSTM-Based Log Anomaly Detection Approach Using Semantic Representation
Fu, Y., Liang, K., & Xu, J. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3289488 · Cited by 47 · see citing works
LogParse: Making Log Parsing Adaptive through Word Classification
Meng, W., Liu, Y., Zaiter, F., Zhang, S., Chen, Y., Zhang, Y., Zhu, Y., Wang, E., Zhang, R., Tao, S., et al. (2020) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn49398.2020.9209681 · Cited by 49 · see citing works
ALchemist: Fusing Application and Audit Logs for Precise Attack Provenance without Instrumentation
Yu, L., Ma, S., Zhang, Z., Tao, G., Zhang, X., Xu, D., Urias, V. E., Han, L., Ciocarlie, G. F., Yegneswaran, V., et al. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24445 · Cited by 47 · see citing works
LogAssist: Assisting Log Analysis Through Log Summarization
Locke, S., Li, H., Chen, T.-H., Shang, W., & Liu, W. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3083715 · Cited by 45 · see citing works
LayerLog: Log sequence anomaly detection based on hierarchical semantics
Zhang, C., Wang, X., Zhang, H., Zhang, J., Zhang, H., Liu, C., & Han, P. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109860 · Cited by 45 · see citing works
Loader: A Log Anomaly Detector Based on Transformer
Xiao, T. Y., Quan, Z., Wang, Z.-J., Le, Y., Du, Y., Liao, X., Li, K., & Li, K. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3280575 · Cited by 43 · see citing works
Tao, S., Meng, W., Cheng, Y., Zhu, Y., Liu, Y., Du, C., Han, T., Zhao, Y., Wang, X., & Yang, H. (2022) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3543146.3543168 · Cited by 45 · see citing works
MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures
Vervaet, A. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00317 · Cited by 47 · see citing works
Building AI Agents for Autonomous Clouds: Challenges and Design Principles
Shetty, M., Chen, Y., Somashekar, G., Ma, M., Simmhan, Y. L., Zhang, X., Mace, J., Vandevoorde, D., Las-Casas, P., Gupta, S. M., et al. (2024) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3698038.3698525 · Cited by 54 · see citing works
PredictDeep: Security Analytics as a Service for Anomaly Detection and Prediction
Elsayed, M. A., & Zulkernine, M. (2020) · IEEE Access · doi:10.1109/access.2020.2977325 · Cited by 41 · see citing works
Improving Log-Based Anomaly Detection with Component-Aware Analysis
Yin, K., Yan, M. L., Xu, L., Xu, Z., Li, Z., Yang, D., & Zhang, X. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00069 · Cited by 41 · see citing works
Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models
Ott, H., Bogatinovski, J., Acker, A., Nedelkoski, S., & Kao, O. (2021) · IEEE/ACM International Workshop on Cloud Intelligence (CloudIntelligence) · doi:10.1109/cloudintelligence52565.2021.00013 · Cited by 42 · see citing works
LogGT: Cross-system log anomaly detection via heterogeneous graph feature and transfer learning
Wang, P., Zhang, X., Cao, Z., Xu, W., & Li, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124082 · Cited by 41 · see citing works
Aarohi: Making Real-Time Node Failure Prediction Feasible
Das, A., Mueller, F., & Rountree, B. (2020) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps47924.2020.00115 · Cited by 38 · see citing works
Log-based software monitoring: a systematic mapping study
Cândido, J., Aniche, M. F., & Arie Van Deursen, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.489 · Cited by 38 · see citing works
On the effectiveness of log representation for log-based anomaly detection
Wu, X., Li, H., & Khomh, F. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10364-1 · Cited by 38 · see citing works
Anomaly Detection using Distributed Log Data: A Lightweight Federated Learning Approach
Guo, Y., Wu, Y., Zhu, Y., Yang, B., & Han, C. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533294 · Cited by 37 · see citing works
An empirical study of the impact of log parsers on the performance of log-based anomaly detection
Fu, Y., Yan, M. L., Xu, Z., Xia, X., Zhang, X., & Yang, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10214-6 · Cited by 38 · see citing works
Wurzenberger, M., Höld, G., Landauer, M., & Skopik, F. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103631 · Cited by 36 · see citing works
KnowLog: Knowledge Enhanced Pre-trained Language Model for Log Understanding
Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., & Xiao, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623304 · Cited by 38 · see citing works
QLLog: A log anomaly detection method based on Q-learning algorithm
Duan, X., Ying, S., Yuan, W., Cheng, H., & Yin, X. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102540 · Cited by 37 · see citing works
Setianto, F., Tsani, E., Sadiq, F., Domalis, G., Tsakalidis, D., & Kostakos, P. (2021) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3487351.3492723 · Cited by 36 · see citing works
Anomaly Detection via Mining Numerical Workflow Relations from Logs
Zhang, B., Zhang, H., Moscato, P., & Zhang, A. (2020) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds51746.2020.00027 · Cited by 34 · see citing works
A study of the performance of general compressors on log files
Yao, K., Li, H., Shang, W., & Hassan, A. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09822-x · Cited by 32 · see citing works
LPV: A Log Parser Based on Vectorization for Offline and Online Log Parsing
Xiao, T. Y., Quan, Z., Wang, Z.-J., Zhao, K., & Liao, X. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00175 · Cited by 32 · see citing works
Research on anomaly detection and real-time reliability evaluation with the log of cloud platform
Wang, B., Hua, Q., Zhang, H., Tan, X. L., Nan, Y., Chen, R., & Shu, X. (2022) · Alexandria Engineering Journal · doi:10.1016/j.aej.2021.12.061 · Cited by 37 · see citing works
OneLog: towards end-to-end software log anomaly detection
Hashemi, S., & Mäntylä, M. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00428-x · Cited by 39 · see citing works
PreLog: A Pre-trained Model for Log Analytics
Le, V.-H., & Zhang, H. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654966 · Cited by 31 · see citing works
Landscape of Automated Log Analysis: A Systematic Literature Review and Mapping Study
Korzeniowski, Ł., & Goczyła, K. (2022) · IEEE Access · doi:10.1109/access.2022.3152549 · Cited by 34 · see citing works
LogLS: Research on System Log Anomaly Detection Method Based on Dual LSTM
Chen, Y., Luktarhan, N., & Lv, D. (2022) · Symmetry · doi:10.3390/sym14030454 · Cited by 32 · see citing works
Paddy: An Event Log Parsing Approach using Dynamic Dictionary
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms47738.2020.9110435 · Cited by 31 · see citing works
An Effective Approach for Parsing Large Log Files
Sedki, I., Hamou‐Lhadj, A., Mohamed, O. A., & Shehab, M. A. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00009 · Cited by 30 · see citing works
Investigating and improving log parsing in practice
Fu, Y., Yan, M. L., Xu, J., Li, J., Liu, Z., Zhang, X., & Yang, D. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558947 · Cited by 34 · see citing works
Improving State-of-the-Art Compression Techniques for Log Management Tools
Yao, K., Sayagh, M., Shang, W., & Hassan, A. E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3069958 · Cited by 29 · see citing works
Framework for automatic detection of anomalies in DevOps
Fawzy, A., Wassif, K. T., & Moussa, H. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.02.010 · Cited by 28 · see citing works
LogGAN: A Sequence-Based Generative Adversarial Network for Anomaly Detection Based on System Logs
Xia, B., Yin, J., Xu, J., & Li, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34637-9_5 · Cited by 29 · see citing works
Kratzke, N. (2022) · Future Internet · doi:10.3390/fi14100274 · Cited by 34 · see citing works
Robust and Transferable Log-based Anomaly Detection
Jia, P., Cai, S., Ooi, B. C., Wang, P., & Xiong, Y. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588918 · Cited by 31 · see citing works
An Assessment of ChatGPT on Log Data
Mudgal, P., & Wouhaybi, R. H. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7587-7_13 · Cited by 33 · see citing works
Studying the use of Java logging utilities in the wild
Chen, B., & Jiang, Z. M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380408 · Cited by 28 · see citing works
- Authors
- Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R.
- Affiliations
- Huawei Technologies (China) (CN)
- Chinese University of Hong Kong (HK)
- Sun Yat-sen University (CN)
- ETH Zurich (CH)
- Southwest Minzu University (CN)
- Venue
- 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 121-130
- OpenAlex
- W2963999143
- Cited by
- 574
- References
- 49
- Citations by year
- 2026: 57 · 2025: 86 · 2024: 108 · 2023: 92 · 2022: 62 · 2021: 89 · 2020: 65 · 2019: 15
- FWCI
- 49.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61332010, 61502401, 61722214)
- Chinese University of Hong Kong (14210717)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- benchmarking, structured data extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
