Benchmarking Large Language Models for Log Analysis, Security, and Interpretation
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLM) continue to demonstrate their utility in a variety of emergent capabilities in different fields. An area that could benefit from effective language understanding in cybersecurity is the analysis of log files. This work explores LLMs with different architectures (BERT, RoBERTa, DistilRoBERTa, GPT-2, and GPT-Neo) that are benchmarked for their capacity to better analyze application and system log files for security. Specifically, 60 fine-tuned language models for log analysis are deployed and benchmarked. The resulting models demonstrate that they can be used to perform log analysis effectively with fine-tuning being particularly important for appropriate domain adaptation to specific log types. The best-performing fine-tuned sequence classification model (DistilRoBERTa) outperforms the current state-of-the-art; with an average F1-Score of 0.998 across six datasets from both web application and system log sources. To achieve this, we propose and implement a new experimentation pipeline (LLM4Sec) which leverages LLMs for log analysis experimentation, evaluation, and analysis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SECURE benchmarks knowledge extraction, understanding, and reasoning across six Industrial Control System datasets [19].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [19]
In the citing paper
“All tested production models exhibit between 26% and 41% successful prompt-injection rates (CyberSecEval 3 against GPT-4 and Llama-3-70B, respectively, as reported in [19,86])”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [19]
In the citing paper
“CyberSecEval (versions 1-3) and CyberSOCEval, summarized in [19,86], benchmark LLM safety properties: insecure coding suggestions, cyberattack helpfulness, prompt-injection resistance, codeinterpreter abuse, and autonomous offensive capability.”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [19]
In the citing paper
“12 patterns are derived from the public CyberSecEval 3 indirect-prompt-injection split [19,86] (rephrased to fit IoT-telemetry observation shapes); 6 patterns adapt indirect-prompt-injection vectors from Greshake et al. [94]; and 7 patterns are author-generated to cover IoT-specific surfaces not present in either source”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [19]
Citations
Cites
Foundations of Machine Learning
(2012) · Boosting · doi:10.7551/mitpress/8291.003.0006 · Cited by 1,074
Drain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · cited 2×
“Conventional log analysis relies on static regular expression parsers (e.g. DRAIN [1]) to extract salient features from the aggregated logs.”
Japanese and Korean voice search
Schuster, M., & Nakajima, K. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6289079 · Cited by 862
What Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620 · cited 1×
“As part of this research, a new benchmarking experimentation pipeline is developed in order to streamline the research conducted in this study.”
LogBERT: Log Anomaly Detection via BERT
Guo, H., Yuan, S., & Wu, X. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534113 · Cited by 454 · cited 1×
“Guo et al. [16] propose an anomaly detection system called LogBERT.”
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361 · cited 2×
“NO RoBERTa Hongchen Guo [18] Drain NO BERT Le [19] Drain NO BERT”
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · cited 1×
“For sequence classification tasks using models like BERT, a dense fully connected linear layer is typically added after the final layer of the model.”
Leveraging deep neural networks for anomaly‐based web application firewall
Vartouni, A. M., Teshnehlab, M., & Kashi, S. S. (2019) · IET Information Security · doi:10.1049/iet-ifs.2018.5404 · Cited by 77
Classification of HTTP Attacks: A Study on the ECML/PKDD 2007 Discovery Challenge
Lawrence Livermore National Lab. (LLNL), L., Gallagher, B., USDOE, & Eliassi‐Rad, T. (2009) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1113394 · Cited by 44 · cited 1×
“NLP enables dynamic feature extraction without the prerequisite of prior knowledge about data structures, enhancing scalability and, crucially, versatility [2, 3].”
Anomaly Detection of System Logs Based on Natural Language Processing and Deep Learning
Wang, M., Xu, L., & Guo, L. (2018) · 2018 4th International Conference on Frontiers of Signal Processing (ICFSP) · doi:10.1109/icfsp.2018.8552075 · Cited by 42 · cited 2×
“Wang et al. [13] propose an intrusion detection system that utilizes word2vec and TF-IDF as a feature extraction method with supervised methods 1 and is benchmarked for classification.”
Research works printed but not linked: 6
- ECML/PKDD: ECML/PKDD 2007 Discovery Challenge (2021). https://gitlab.fing.edu.uy/gsi/web-application-attacks-datasets/-/tree/master/ecml_pkdd
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. CoRR (2017). arXiv:abs/1706.03762
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR (2019). arXiv:abs/1907.11692
- Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019). ArXiv:abs/1910.01108
- Kokalj, E., Škrlj, B., Lavrač, N., Pollak, S., Robnik-Šikonja, M.: BERT meets Shapley: extending SHAP explanations to transformer-based classifiers. In: Toivonen, H., Boggia, M. (eds.) Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, 2021, pp. 16–21. Association for Computational Linguistics (2021). https://aclanthology.org/2021.hackashop-1.3
- Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
Cited by
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“They emphasizes the potential of LLMs to revolutionize the field of email security and suggests that these models can be valuable tools for improving the security of information systems.”
Large Language Models for Cyber Security: A Systematic Literature Review
Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769676 · Cited by 189
“The dominance of decoder-only LLMs in cybersecurity research aligns with the broader trends in the LLM community.”
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
Application of Large Language Models in Cybersecurity: A Systematic Literature Review
Hasanov, I., Virtanen, S., Hakkala, A., & Isoaho, J. (2024) · IEEE Access · doi:10.1109/access.2024.3505983 · Cited by 89
“Karlsen et al. [205] benchmark several LLMs for log analysis and security using the LLM4Sec pipeline.”
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69
Metadata
- Authors
- Karlsen, E., Luo, X., Zincir-Heywood, N., & Heywood, M.
- Affiliations
- Dalhousie University (CA)
- Oklahoma State University (US)
- Corresponding author
- Karlsen, Egil
- Venue
- Journal of Network and Systems Management
- Published
- 2024-06-13
- Volume
- 32
- Issue
- 3
- ISSN
- 1064-7570 (ISSN-L)
- 1573-7705
- OpenAlex
- W4399625707
- Cited by
- 71
- References
- 25
- Citations by year
- 2026: 31 · 2025: 35 · 2024: 5
- FWCI
- 11.85
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-nd
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Journal of Network and Systems Management (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, log analysis, cybersecurity, log classification, fine-tuning, DistilRoBERTa, BERT, RoBERTa, GPT-2, GPT-Neo, system logs, benchmarking, domain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking Large Language Models for Log Analysis, Security, and Interpretation. https://app.cochat.ai/papers/10.1007/s10922-024-09831-x
- The paper
- Karlsen, E., Luo, X., Zincir-Heywood, N., & Heywood, M. (2024). Benchmarking Large Language Models for Log Analysis, Security, and Interpretation. Journal of Network and Systems Management. https://doi.org/10.1007/s10922-024-09831-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.