CoChat
Sign Up
PreprintCited by 14Software System Performance and ReliabilityAnomaly Detection Techniques and ApplicationsSoftware Engineering Research

LogLLM: Log-based Anomaly Detection Using Large Language Models

Open access
Guan, Wei, Cao, Jian, Qian, Shiyou, Gao, Jianqi, Ouyang, Chun·2024·arXiv
DOI 10.48550/arxiv.2411.08561arXiv 2411.08561
Open this paper in CoChat — free accountOpen PDF

Abstract

Software systems often record important runtime information in logs to help with troubleshooting. Log-based anomaly detection has become a key research area that aims to identify system issues through log data, ultimately enhancing the reliability of software systems. Existing methods often fall short in capturing the semantic information, typically expressed in natural language, or the sequential dependencies inherent in log sequences. In this paper, we propose LogLLM, a framework that enables collaboration between heterogeneous LLMs for log-based anomaly detection. LogLLM exploits the complementary capabilities of different LLM architectures: a Transformer encoder-based LLM is employed to extract fine-grained semantic vectors from individual log messages, while a Transformer decoder-based LLM is utilized to model sequential dependencies and generate anomaly detection decisions. To enable effective collaboration between these heterogeneous LLMs, we introduce a learnable projector to align their vector representation spaces. Furthermore, we design a progressive three-stage training strategy to optimize the collaboration between heterogeneous LLMs by gradually aligning their representations and adapting them to log anomaly detection. Unlike conventional methods that require log parsers to extract templates, LogLLM preprocesses log messages with regular expressions, streamlining the entire process. Experimental results on four public real-world datasets demonstrate that LogLLM outperforms state-of-the-art methods, achieving an average F$_1$-score improvement of 6.6% over the strongest existing approach. Further analyses provide insights into the effectiveness of the key components of the model architecture and progressive three-stage training strategy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.08561

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Class imbalance and the limited availability of annotated training samples restrict the applicability of conventional supervised machine learning techniques.

    “Compounding the issue are problems like class imbalance -since anomalies are rare by definition -and the limited availability of annotated training samples, which restricts the applicability of conventional supervised machine learning techniques [17,18].”

    From Log anomaly detection in AIOps: A real-world implementation using Large Language Models (2026) · cites this paper as [18]

    The Training section says supervised anomaly detection needs labeled normal and anomalous samples and that class imbalance can bias training; the Traditional Deep Learning section also describes a method addressing insufficient labeling. But the claim’s explanation that anomalies are rare by definition is too broad: the Minority Class Oversampling section reports a dataset in which anomalous samples outnumber normal ones. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • LogLLM evaluated on HDFS and BGL datasets achieved F1-scores of 0.997 and 0.916, respectively.

    “More recently, LogLLM [18] evaluated on HDFS and BGL datasets achieved F1-scores of 0.997 and 0.916, respectively.”

    From Log anomaly detection in AIOps: A real-world implementation using Large Language Models (2026) · cites this paper as [18]

  • LogLLM combines an encoder (BERT) and a decoder (Llama), aligning log semantic sequences through a projection layer to achieve end-to-end anomaly detection.

    “LogLLM[38] combines an encoder (BERT) and a decoder (Llama), aligning log semantic sequences through a projection layer to achieve end-to-end anomaly detection.”

    From From Correlation to Intervention: A Survey on Log-Based Causal Analysis for Security and AIOps (2026) · cites this paper as [38]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 56

  1. R. S. Kazemzadeh and H.-A. Jacobsen, "Reliable and highly available distributed publish/subscribe service," in 2009 28th IEEE International Symposium on Reliable Distributed Systems. IEEE, 2009, pp. 41-50.
  2. E. Bauer and R. Adams, Reliability and availability of cloud computing. John Wiley & Sons, 2012.
  3. V.-H. Le and H. Zhang, "Log-based anomaly detection without log pars- ing," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 492-504.
  4. W. Guan, J. Cao, H. Zhao, Y. Gu, and S. Qian, "Survey and benchmark of anomaly detection in business processes," IEEE Transactions on Knowledge and Data Engineering, pp. 1-23, 2024.
  5. V.-H. Le and H. Zhang, "Log-based anomaly detection with deep learning: How far are we?" in Proceedings of the 44th international conference on software engineering, 2022, pp. 1356-1367.
  6. J. Qi, S. Huang, Z. Luan, S. Yang, C. Fung, H. Yang, D. Qian, J. Shang, Z. Xiao, and Z. Wu, "Loggpt: Exploring chatgpt for log-based anomaly detection," in 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys). IEEE, 2023, pp. 273-280.
  7. M. Du, F. Li, G. Zheng, and V. Srikumar, "Deeplog: Anomaly detection and diagnosis from system logs through deep learning," in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, 2017, pp. 1285-1298.
  8. W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun et al., "Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs." in IJCAI, vol. 19, no. 7, 2019, pp. 4739-4745.
  9. L. Zhang, W. Li, Z. Zhang, Q. Lu, C. Hou, P. Hu, T. Gui, and S. Lu, "Logattn: Unsupervised log anomaly detection with an autoencoder based attention mechanism," in International conference on knowledge science, engineering and management. Springer, 2021, pp. 222-235.
  10. M. Catillo, A. Pecchia, and U. Villano, "Autolog: Anomaly detection by deep autoencoding of system logs," Expert Systems with Applications, vol. 191, p. 116263, 2022.
  11. Y. Xie and K. Yang, "Log anomaly detection by adversarial autoencoders with graph feature fusion," IEEE Transactions on Reliability, 2023.
  12. X. Zhang, X. Chai, M. Yu, and D. Qiu, "Anomaly detection model for log based on lstm network and variational autoencoder," in 2023 4th International Conference on Information Science, Parallel and Distributed Systems (ISPDS). IEEE, 2023, pp. 239-244.
  13. X. Duan, S. Ying, W. Yuan, H. Cheng, and X. Yin, "A generative adversarial networks for log anomaly detection." Comput. Syst. Sci. Eng., vol. 37, no. 1, pp. 135-148, 2021.
  14. Z. He, Y. Tang, K. Zhao, J. Liu, and W. Chen, "Graph-based log anomaly detection via adversarial training," in International Symposium on Dependable Software Engineering: Theories, Tools, and Applications. Springer, 2023, pp. 55-71.
  15. C. Zhang, X. Wang, H. Zhang, J. Zhang, H. Zhang, C. Liu, and P. Han, "Layerlog: Log sequence anomaly detection based on hierarchical se- mantics," Applied Soft Computing, vol. 132, p. 109860, 2023.
  16. S. Hashemi and M. Mäntylä, "Onelog: towards end-to-end software log anomaly detection," Automated Software Engineering, vol. 31, no. 2, p. 37, 2024.
  17. S. Lu, X. Wei, Y. Li, and L. Wang, "Detecting anomaly in big data system logs using convolutional neural network," in 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech). IEEE, 2018, pp. 151-158.
  18. X. Zhang, Y. Xu, Q. Lin, B. Qiao, H. Zhang, Y. Dang, C. Xie, X. Yang, Q. Cheng, Z. Li et al., "Robust log-based anomaly detection on unstable log data," in Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, 2019, pp. 807-817.
  19. Y. Xie, H. Zhang, and M. A. Babar, "Loggd: Detecting anomalies from system logs with graph neural networks," in 2022 IEEE 22nd International conference on software quality, reliability and security (QRS). IEEE, 2022, pp. 299-310.
  20. Z. Zhao, W. Niu, X. Zhang, R. Zhang, Z. Yu, and C. Huang, "Trine: Syslog anomaly detection with three transformer encoders in one gen- erative adversarial network," Applied Intelligence, vol. 52, no. 8, pp. 8810-8819, 2022.
  21. L. Yang, J. Chen, Z. Wang, W. Wang, J. Jiang, X. Dong, and W. Zhang, "Semi-supervised log-based anomaly detection via probabilistic label estimation," in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1448-1460.
  22. S. Hochreiter, "Long short-term memory," Neural Computation MIT- Press, 1997.
  23. A. Vaswani, "Attention is all you need," Advances in Neural Information Processing Systems, 2017.
  24. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  25. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., "The llama 3 herd of models," arXiv preprint arXiv:2407.21783, 2024. · arXiv
  26. T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai et al., "Chatglm: A family of large language models from glm-130b to glm-4 all tools," arXiv preprint arXiv:2406.12793, 2024. · arXiv
  27. W. Guan, J. Cao, J. Gao, H. Zhao, and S. Qian, "Dabl: Detecting semantic anomalies in business processes using large language models," arXiv preprint arXiv:2406.15781, 2024. · arXiv
  28. Y. Liu, S. Tao, W. Meng, F. Yao, X. Zhao, and H. Yang, "Logprompt: Prompt engineering towards zero-shot and interpretable log analysis," in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 364-365.
  29. C. Egersdoerfer, D. Zhang, and D. Dai, "Early exploration of using chatgpt for log-based anomaly detection on parallel file systems logs," in Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing, 2023, pp. 315-316.
  30. J. Pan, W. S. Liang, and Y. Yidi, "Raglog: Log anomaly detection using retrieval augmented generation," in 2024 IEEE World Forum on Public Safety Technology (WFPST). IEEE, 2024, pp. 169-174.
  31. H. Guo, S. Yuan, and X. Wu, "Logbert: Log anomaly detection via bert," in 2021 international joint conference on neural networks (IJCNN). IEEE, 2021, pp. 1-8.
  32. Y. Lee, J. Kim, and P. Kang, "Lanobert: System log anomaly detection based on bert masked language model," Applied Soft Computing, vol. 146, p. 110689, 2023.
  33. Y. Lin, H. Deng, and X. Li, "Fastlogad: Log anomaly detection with mask-guided pseudo anomaly generation and discrimination," arXiv preprint arXiv:2404.08750, 2024. · arXiv
  34. C. Almodovar, F. Sabrina, S. Karimi, and S. Azad, "Logfit: Log anomaly detection using fine-tuned language models," IEEE Transactions on Network and Service Management, 2024.
  35. S. Chen and H. Liao, "Bert-log: Anomaly detection for system logs based on pre-trained language model," Applied Artificial Intelligence, vol. 36, no. 1, p. 2145642, 2022.
  36. J. L. Adeba, D.-H. Kim, and J. Kwak, "Sarlog: Semantic-aware robust log anomaly detection via bert-augmented contrastive learning," IEEE Internet of Things Journal, 2024.
  37. Y. Fu, K. Liang, and J. Xu, "Mlog: Mogrifier lstm-based log anomaly detection approach using semantic representation," IEEE Transactions on Services Computing, vol. 16, no. 5, pp. 3537-3549, 2023.
  38. G. No, Y. Lee, H. Kang, and P. Kang, "Training-free retrieval-based log anomaly detection with pre-trained language model considering token- level information," Engineering Applications of Artificial Intelligence, vol. 133, p. 108613, 2024.
  39. F. Hadadi, Q. Xu, D. Bianculli, and L. Briand, "Anomaly detection on unstable logs with gpt models," arXiv preprint arXiv:2406.07467, 2024. · arXiv
  40. M. Burtsev, M. Reeves, and A. Job, "The working limitations of large language models," MIT Sloan Management Review, vol. 65, no. 2, pp. 8-10, 2024.
  41. A. Joulin, "Fasttext. zip: Compressing text classification models," arXiv preprint arXiv:1612.03651, 2016. · arXiv
  42. J. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
  43. Y. Liu, "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  44. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, "Spanbert: Improving pre-training by representing and predicting spans," Transactions of the association for computational linguistics, vol. 8, pp. 64-77, 2020.
  45. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., "Retrieval- augmented generation for knowledge-intensive nlp tasks," Advances in Neural Information Processing Systems, vol. 33, pp. 9459-9474, 2020.
  46. J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, "Tools and benchmarks for automated log parsing," in 2019 IEEE/ACM 41st In- ternational Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2019, pp. 121-130.
  47. P. He, J. Zhu, S. He, J. Li, and M. R. Lyu, "An evaluation study on log parsing and its use in log mining," in 2016 46th annual IEEE/IFIP international conference on dependable systems and networks (DSN). IEEE, 2016, pp. 654-661.
  48. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, "Online system problem detection by mining patterns of console logs," in 2009 ninth IEEE international conference on data mining. IEEE, 2009, pp. 588- 597.
  49. A. Oliner and J. Stearley, "What supercomputers say: A study of five system logs," in 37th annual IEEE/IFIP international conference on dependable systems and networks (DSN'07). IEEE, 2007, pp. 575- 584.
  50. P. He, J. Zhu, Z. Zheng, and M. R. Lyu, "Drain: An online log parsing approach with fixed depth tree," in 2017 IEEE international conference on web services (ICWS). IEEE, 2017, pp. 33-40.
  51. M. Du and F. Li, "Spell: Streaming parsing of system event logs," in 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 2016, pp. 859-864.
  52. V.-H. Le and H. Zhang, "Log parsing with prompt-based few-shot learning," in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2438-2449.
  53. J. S. Bridle, "Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition," in Neurocomputing: Algorithms, architectures and applications. Springer, 1990, pp. 227-236.
  54. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "Qlora: Efficient finetuning of quantized llms," Advances in Neural Information Processing Systems, vol. 36, 2024.
  55. I. Loshchilov, "Decoupled weight decay regularization," arXiv preprint arXiv:1711.05101, 2017. · arXiv
  56. J. Zhu, S. He, P. He, J. Liu, and M. R. Lyu, "Loghub: A large collection of system log datasets for ai-driven log analytics," in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2023, pp. 355-366.

Cited by

Sorted by citations

Related

Authors
Guan, W., Cao, J., Qian, S., Gao, J., & Ouyang, C.
Affiliations
  • Tianjin University (CN)
  • Shanghai Jiao Tong University (CN)
  • Shanghai University (CN)
  • Queensland University of Technology (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
1-16
ISSN
0098-5589 (ISSN-L)
DOI
10.48550/arxiv.2411.08561
arXiv
2411.08561
OpenAlex
W4404407704
Cited by
11
References
0
Citations by year
2026: 8 · 2025: 3
FWCI
30.55
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Transactions on Software Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
log anomaly detection, large language models, software logs, BERT, llamas, log parsing, software reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review