Transformer-based LLMs in Cybersecurity: An in-depth Study on Log Anomaly Detection and Conversational Defense Mechanisms
The DOI matches its Crossref record; 16 of the 17 references it cites were found in a registry or an index.
Abstract
With the advancement of conversational AI and Large Language Models (LLMs), interactive chatbots are emerging as pivotal assets for connecting with users across various sectors, enabling various capabilities and functions. However, their potential in the cybersecurity domain remains largely untapped. This article introduces a novel method to enhance chatbot performance by incorporating anomaly detection features. Our chatbot uses advanced GPT-3 models and rule-based logic to identify and extract unusual patterns and deviations within logs, making it more proficient in detecting anomalies. We present the architecture and methodology behind our anomaly detection system, showcasing its effectiveness in real-world scenarios. Combining machine learning and domain expertise, our chatbot sets a new standard in interactive, anomaly-aware conversational agents. Our anomaly detection classifier was able to achieve more than 99% of accuracy by illustrating its robust performance in accurately identifying and flagging outliers or unusual patterns in log file data. We also compared the performance of GPT-3 models with other LLMs: BERT, DistilBERT, and ALBERT. Our findings concluded that GPT-3 models consistently outperform all the other LLM models and exhibit significantly higher performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The first is the emergence of large language models (LLMs) capable of zero-shot reasoning over heterogeneous textual artifacts-packet captures rendered as text, system logs, threat-intelligence reports, MITRE ATT&CK technique descriptions, CVE narratives-without bespoke feature engineering”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [16]
In the citing paper
“Hybrid IDS architectures that combine line-rate ML detectors with gateway-resident LLMs for triage are gaining traction because they balance latency (tens of milliseconds) with reasoning capacity [16,68,74,87].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [16]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“The BERT model, developed by Google AI [5], stands as a state-of-the-art transformer-based model that is pre-trained bidirectionally on an extensive textual dataset.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“The BERT model, developed by Google AI [5], stands as a state-of-the-art transformer-based model that is pre-trained bidirectionally on an extensive textual dataset.”
MethodDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · cited 1×
“2) DistilBERT DistilBERT is perceived as a streamlined and swifter counterpart to BERT, birthed by Hugging Face [11].”
BackgroundALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 1×
“3) ALBERT (A Lite BERT) ALBERT, developed by Google AI [9], emerged as a solution to address BERT’s limitations, specifically targeting a diminished model size and augmented training efficacy.”
MethodExperience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R. (2016) · 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2016.21 · Cited by 593 · cited 1×
“[7].”
Loghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302 · cited 1×
“To support this training, we utilize a comprehensive and annotated open-source dataset [17] from prior research on anomaly detection [18], establishing a robust groundwork for detecting anomalies.”
Evaluation of Machine Learning Algorithms for Anomaly Detection
Elmrabit, N., Zhou, F., Li, F., & Zhou, H. (2020) · 2020 International Conference on Cyber Security and Protection of Digital Services (Cyber Security) · doi:10.1109/cybersecurity49315.2020.9138871 · Cited by 212 · cited 1×
“Elmrabit et al. [6] conducted a comprehensive comparison of various machine learning algorithms, including prominent algorithms like Long Short-Term Memory (LSTM), Decision Tree (DT), Random Forest (RF), Deep Neural Network (DNN), etc.”
Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection
Bertero, C., Roy, M., Sauvanaud, C., & Tredan, G. (2017) · 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2017.43 · Cited by 139 · cited 1×
“In [2], the authors employed the word2vec model to produce word embeddings for log data.”
Dynamic log file analysis: An unsupervised cluster evolution approach for anomaly detection
Landauer, M., Wurzenberger, M., Skopik, F., Settanni, G., & Peter Filzmoser, A. (2018) · Computers & Security · doi:10.1016/j.cose.2018.08.009 · Cited by 89 · cited 2×
“The importance of identifying anomalies within logs has been underscored in various studies [10, 13, 4].”
BackgroundMachine learning to detect anomalies in web log analysis
Cao, Q., Qiao, Y., & Lyu, Z. (2017) · 2017 3rd IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/compcomm.2017.8322600 · Cited by 55 · cited 2×
“The importance of identifying anomalies within logs has been underscored in various studies [10, 13, 4].”
Research works printed but not linked: 3
- He, 2020, Loghub: A large collection of system log datasets towards automated log analytics, arXiv preprint arXiv:2008.06448 · arXiv
- Saxena, 2021, IoT Dynamic Log File Analysis: Security Approach for Anomaly Detection In Multi Sensor Environment, International research journal of modernization in engineering technology & Science, 3, 2582
- 10.7717/peerj.12726/table-1 · doi:10.7717/peerj.12726/table-1
Cited by
Kheddar, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103347 · Cited by 161
A Survey on XAI for 5G and Beyond Security: Technical Aspects, Challenges and Research Directions
Senevirathna, T., La, V. H., Marchal, S., Siniarski, B., Liyanage, M., & Wang, S. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3437248 · Cited by 148
“Furthermore, LLM-based penetrating testing tools (e.g.: PentestGPT [169]), anomaly detection tools (e.g., PAC-GPT [170], LogBERT [171], LogBot [172]), and threat detection tools (e.g., TSTEM [173], Cyber Sentinel [174]) The rise of this new addition to the security stack of 5G and beyond begs the…”
Balasubramanian, P., Liyana, S., Sankaran, H., Sivaramakrishnan, S., Pusuluri, S., Pirttikangas, S., & Peltonen, E. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11338-z · Cited by 78
Self-citationLLMs are one-shot URL classifiers and explainers
Rashid, F., Ranaweera, N., Doyle, B., & Seneviratne, S. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.111004 · Cited by 40
CYGENT: A cybersecurity conversational agent with log summarization powered by GPT-3
Balasubramanian, P., Seby, J., & Kostakos, P. (2024) · International Conference on Artificial Intelligence for Internet of Things (AIIoT) · doi:10.1109/aiiot58432.2024.10574658 · Cited by 26
“As part of feature development, in our previous paper [11] we developed an anomaly detection feature for the CYGENT framework which uses fine tuned GPT3 models for detection of anomalies in the log data.”
MethodSelf-citation
Metadata
- Authors
- Balasubramanian, P., Seby, J., & Kostakos, P.
- Affiliations
- University of Oulu (FI)
- Venue
- 2023 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2023-12-15
- Pages
- 3590-3599
- OpenAlex
- W4391094158
- Cited by
- 22
- References
- 17
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 6
- FWCI
- 6.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- Funders
- European Commission
- Academy of Finland
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Keywords
- log anomaly detection, chatbots, large language models, GPT-3, cybersecurity, transformer models, BERT, DistilBERT, ALBERT, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Transformer-based LLMs in Cybersecurity: An in-depth Study on Log Anomaly Detection and Conversational Defense Mechanisms. https://app.cochat.ai/papers/10.1109/bigdata59044.2023.10386976
- The paper
- Balasubramanian, P., Seby, J., & Kostakos, P. (2023). Transformer-based LLMs in Cybersecurity: An in-depth Study on Log Anomaly Detection and Conversational Defense Mechanisms. 2023 IEEE International Conference on Big Data (BigData). https://doi.org/10.1109/bigdata59044.2023.10386976
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.