Impact of log parsing on deep learning-based anomaly detection
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Software systems log massive amounts of data, recording important runtime information. Such logs are used, for example, for log-based anomaly detection, which aims to automatically detect abnormal behaviors of the system under analysis by processing the information recorded in its logs. Many log-based anomaly detection techniques based on deep learning models include a pre-processing step called log parsing. However, understanding the impact of log parsing on the accuracy of anomaly detection techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted. In this paper, we report on a comprehensive empirical study on the impact of log parsing on anomaly detection accuracy, using 13 log parsing techniques, seven anomly detection techniques (five based on deep learning and two based on traditional machine learning) on three publicly available log datasets. Our empirical results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and anomaly detection accuracy, regardless of the metric used for measuring log parsing accuracy. Moreover, we experimentally confirm existing theoretical results showing that it is a property that we refer to as distinguishability in log parsing results—as opposed to their accuracy—that plays an essential role in achieving accurate anomaly detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research has focused on tasks such as anomaly detection [90, 237], incident classification [36, 68], and root cause analysis [228, 247].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [90]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“…learning models, we included two representative traditional machine learning models, namely Support Vector Machine (SVM) (Hearst et al. 1998) and Random Forest(RF)Breiman(2001) 2 sincetheyareknownfortheireffectivenessinanomalydetection tasks on the HDFS dataset (Wu et al. 2023; Jia et al. 2023).”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“…log representation techniques, such as FastText (Joulin et al. 2016), Word2Vec (Mikolov et al. 2013), TF-IDF (Salton and Buckley 1988) and BERT (Devlin et al. 2018), used to convert textual log data into numerical feature vectors for machine learning algorithms, such as Support Vector Machine,…”
BackgroundMethodEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“They considered different log representation techniques, such as FastText (Joulin et al. 2016), Word2Vec (Mikolov et al. 2013), TF-IDF (Salton and Buckley 1988) and BERT (Devlin et al. 2018), used to convert textual log data into numerical feature vectors for machine learning algorithms, such as Support Vector Machine…”
MethodTerm-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 2×
“They considered different log representation techniques, such as FastText (Joulin et al. 2016), Word2Vec (Mikolov et al. 2013), TF-IDF (Salton and Buckley 1988) and BERT (Devlin et al. 2018), used to convert textual log data into numerical feature vectors for machine learning algorithms, such as Support Vector Machine…”
MethodHearst, M. A., Dumais, S. T., Osuna, E., Platt, J., & Scholkopf, B. (1998) · IEEE Intelligent Systems and their Applications · doi:10.1109/5254.708428 · Cited by 7,243 · cited 8×
“In Table 6, except for PLELog, SVM, and RF, the values for all anomaly detection techniques are identical due to the reasons explained in the last paragraph of Section 5.1.”
MethodInfluentialOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_16 · Cited by 2,915 · cited 1×
“To see if the differences between R dst and R ind are significant, we applied the non-parametric L og R obu s t ( F 1 ) Wilcoxon signed rank test (Wilcoxon 1992) for paired samples to the F1 scores of R dst and R ind , for each of the seven anomaly detection techniques and the two datasets.”
MethodDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835 · cited 12×
“…would be enough in some cases (e.g., detecting anomalies based on the sequence of template IDs without considering the content of the templates (Du et al. 2017)), correctly identified templates (i.e., templates identical to the corresponding oracle ones) matter when the fixed parts of templates…”
BackgroundMethodInfluentialDrain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · cited 7×
“…various elements of each variable part with a special character “ <*> ”; this representation is widely used in log-based analyses (e.g., log parsing (He et al. 2017; Jiang et al. 2008), anomaly detection (Zhang et al. 2019; Du et al. 2017), and log-based testing (Elyasov 2012; Jeong et al.…”
BackgroundMethodInfluentialFastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887 · cited 4×
“Furthermore, this is the first work to empirically demonstrate the importance of distinguishability after the theoretical framework proposed by Shin et al. (2021).”
BackgroundMethodInfluential
Research works printed but not linked: 7
- Ali Abd Al-Hameed K (2022) Spearman’s correlation coefficient in statistical analysis. Int J Nonlinear Anal Appl 13(1):3249–3255
- He S, Zhu J, He P, Lyu MR (2020) Loghub: A large collection of system log datasets towards automated log analytics. arXiv:2008.06448https://arxiv.org/pdf/2008.06448.pdf
- Huang C, Guan H, Jiang A, Zhang Y, Spratling M, Wang YF (2022) Registration based few-shot anomaly detection. European conference on computer vision. Springer, Springer, New York, NY, USA, pp 303–319
- Meng W, Liu Y, Zhu Y, Zhang S, Pei D, Liu Y, Chen Y, Zhang R, Tao S, Sun P et al (2019) Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. IJCAI, ACM, New York, NY, USA 19:4739–4745
- Oliner A, Stearley J (2007) What supercomputers say: A study of five system logs. 37th annual IEEE/IFIP international conference on dependable systems and networks (DSN’07), IEEE, IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). Edinburgh, UK, pp 575–584
- Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine learning in python. J Mach Learn Res 12:2825–2830
- Vaarandi R, Pihelgas M (2015) Logcluster - a data clustering and pattern mining algorithm for event logs. In: 2015 11th International conference on network and service management (CNSM), IEEE, Los Alamitos, CA, USA, pp 1–7. https://doi.org/10.1109/CNSM.2015.7367331 · DOI · Paper page
Cited by
A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?
Jiang, Z., Liu, J., Huang, J., Li, Y., Huo, Y., Gu, J., Chen, Z., Zhu, J., & Lyu, M. R.-T. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652123 · Cited by 172
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Ma, Z., Chen, A. R., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639150 · Cited by 92
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 90
Self-citationArtificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 51
A comprehensive study of machine learning techniques for log-based anomaly detection
Ali, S., Boufaied, C., Bianculli, D., Branco, P., & Briand, L. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10669-3 · Cited by 48
Self-citation
Metadata
- Authors
- Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C.
- Affiliations
- University of Luxembourg (LU)
- University of Sheffield (GB)
- University of Ottawa (CA)
- Corresponding author
- Khan, Zanis Ali
- Venue
- Empirical Software Engineering
- Published
- 2024-08-17
- Volume
- 29
- Issue
- 6
- Pages
- 139-139
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- PMC
- PMC11330418
- OpenAlex
- W4401667316
- Cited by
- 49
- References
- 42
- Citations by year
- 2026: 23 · 2025: 25 · 2024: 1
- FWCI
- 8.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion, cc-by)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion, cc-by)
- Figshare (submittedVersion, cc-by-nc-sa)
- University of Limerick Institutional Repository (University of Limerick) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (cc-by)
- University of Limerick Institutional Repository (University of Limerick)
- Funders
- Fonds National de la Recherche Luxembourg (C22/IS/17373407/LOGODOR)
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- log parsing, log anomaly detection, deep learning, distinguishability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Impact of log parsing on deep learning-based anomaly detection. https://app.cochat.ai/papers/10.1007/s10664-024-10533-w
- The paper
- Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). Impact of log parsing on deep learning-based anomaly detection. Empirical Software Engineering. https://doi.org/10.1007/s10664-024-10533-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.