Log-based anomaly detection with deep learning: how far are we?
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
Software-intensive systems produce logs for troubleshooting purposes. Recently, many deep learning models have been proposed to automatically detect system anomalies based on log data. These models typically claim very high detection accuracy. For example, most models report an F-measure greater than 0.9 on the commonly-used HDFS dataset. To achieve a profound understanding of how far we are from solving the problem of log-based anomaly detection, in this paper, we conduct an in-depth analysis of five state-of-the-art deep learning-based models for detecting system anomalies on four public log datasets. Our experiments focus on several aspects of model evaluation, including training data selection, data grouping, class distribution, data noise, and early detection ability. Our results point out that all these aspects have significant impact on the evaluation, and that all the studied models do not always work well. The problem of log-based anomaly detection has not been solved yet. Based on our findings, we also suggest possible future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Extensive machine learning and deep learning-based techniques have been proposed in research for automated log anomaly detection [9], [10], with the aim of reducing manual analysis costs.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the adoption of LLM agents in the AIOps field, especially with noisy and lengthy data, remains limited [21,28].”
From RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models (2024) · cites this paper as [21]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“For example, Logsy [39] utilizes the Transformer network [46] to detect anomalies from log data.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“PLELog extracts the semantic vectors of log templates by using a pre-trained Glovemodel [41].”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“Recurrent Neural Networks (RNNs), including its variants Long Short-Term Memory (LSTM) [21] and Gated Recurrent Units (GRUs) [9], are neural networks designed to handle the sequential inputs with arbitrary length.”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · cited 1×
“In [33], a CNN model originally designed for sentence classification [25] is applied to log-based anomaly detection.”
MethodLearning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · cited 1×
“Recurrent Neural Networks (RNNs), including its variants Long Short-Term Memory (LSTM) [21] and Gated Recurrent Units (GRUs) [9], are neural networks designed to handle the sequential inputs with arbitrary length.”
Backgroundhdbscan: Hierarchical density based clustering
McInnes, L., Healy, J., & Astels, S. (2017) · The Journal of Open Source Software · doi:10.21105/joss.00205 · Cited by 2,833 · cited 2×
“PLELog adopts a clustering method (i.e., HDBSCAN [35]) to probabilistically estimate the labels of unlabeled log sequences.”
BackgroundBeyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11941439_114 · Cited by 1,974 · cited 1×
“Specificity [43, 49], which is the percentage of log sequences that are correctly identified normal over all real normal sequences, can measure the probability of false alarms.”
BackgroundDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835 · cited 10×
“To answer the above questions, we conduct a systematic evaluation of five representative deep learning models for log-based anomaly detection (including DeepLog [12], LogAnomaly [36], PLELog [48], LogRobust [54], and CNN [33]) on four datasets (including HDFS, BGL, Spirit, and Thunderbird), under controlled experiment…”
BackgroundMethodInfluentialDetecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 1×
“Over the years, many data-driven methods have been proposed to automatically detect system anomalies by analyzing log data [6, 7, 12, 29, 47, 50, 54].”
Research works printed but not linked: 2
- [21] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735-1780. -
- [31] XuLiu, Weiyou Liu, Xiaoqiang Di, Jinqing Li, Binbin Cai, Weiwu Ren, and Huamin Yang. 2021. LogNADS: Network anomaly detection scheme based on semantic representation. Future Generation Computer Systems (2021). -
Cited by
Deep learning for anomaly detection in log data: A survey
Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100470 · Cited by 202
“tions of evaluation results, it is recommended to also compute metrics that are more robust against class imbalance, such as the specificity or true negative rate [10].”
BackgroundMethodInfluentialLog Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129
“Typical applications of log analysis include anomaly detection [1], [54], [55], [56], failure prediction [7], [8], root cause analysis [5], [6], etc.”
MethodSelf-citationLILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Jiang, Z., Liu, J., Chen, Z., Li, Y., Huang, J., Huo, Y., He, P., Gu, J., & Lyu, M. R. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643733 · Cited by 123
“LILAC consists of two main components, the ICL-enhanced parser and the adaptive parsing cache .”
LogFiT: Log Anomaly Detection Using Fine-Tuned Language Models
Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3358730 · Cited by 111
“Consequently, Machine Learning (ML) and Deep Learning (DL)-based solutions have been proposed to automatically detect anomalies from system log data, thereby reducing the burden on human operators [4], [5], [6].”
BackgroundLogGPT: Exploring ChatGPT for Log-Based Anomaly Detection
Qi, J., Huang, S., Luan, Z., Yang, S., Fung, C., Yang, H., Qian, D., Shang, J., Xiao, Z., & Wu, Z. (2023) · 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys) · doi:10.1109/hpcc-dss-smartcity-dependsys60770.2023.00045 · Cited by 102
“We believe this is still due to the abnormal diversity, BGL (143 anomaly) has more diversity of abnormal than Spirit (121 anomaly).”
MethodInfluential
Metadata
- Authors
- Le, V.-H., & Zhang, H.
- Affiliations
- University of Newcastle Australia (AU)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1356-1367
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4226128225
- Cited by
- 257
- References
- 51
- Citations by year
- 2026: 59 · 2025: 88 · 2024: 57 · 2023: 49 · 2022: 4
- FWCI
- 47.97
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Funders
- Australian Research Council?s Discovery Projects (DP200102940)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- log anomaly detection, deep learning models, model evaluation, training data selection, data grouping, class distribution, noisy data, early detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.