Exploring LLM-Based Agents for Root Cause Analysis
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team’s specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized Large Language Models (LLMs) to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at a large IT corporation. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“A recent development in LM research has been the rise of LMs augmented with the ability to reason and use tools, or Augmented Language Models (ALMs) [13, 20, 27].”
The use of MMR, diversity-based reranking for reordering documents and producing summaries
Carbonell, J., & Goldstein, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291025 · Cited by 2,329
Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., et al. (2024) · Proceedings of the Nineteenth European Conference on Computer Systems · doi:10.1145/3627703.3629553 · Cited by 190
Self-citationExpeL: LLM Agents Are Experiential Learners
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2023) · arXiv · doi:10.48550/arxiv.2308.10144 · Cited by 179
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
Self-citationWebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1508 · Cited by 107
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31
Research works printed but not linked: 21
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, Ann Arbor, Michigan, ( June 2005 ), 65-72. https://aclanthology.org/W05-0909.
- Chetan Bansal, Sundararajan Renganathan, Ashima Asudani, Olivier Midy, and Mathru Janakiraman. 2019. Decaf: diagnosing and triaging performance issues in large-scale cloud services. CoRR, abs/ 1910.05339. http://arxiv.org/abs / 1910.05339 arXiv: 1910.05339. · arXiv
- Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304. 05128. http://ar xiv. org/abs/2304.05128.
- In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 1701-1716. https://www.usenix.org/conference/nsdi23/presentation /chen-yinfang.
- Proceedings of Machine Learning Research 202 10764-10799. Andreas Krause Emma Brunskill Kyunghyun Cho Barbara Engelhardt Sivan Sabato and Jonathan Scarlett (Eds.) https://proceedings.mlr.press/v202/gao23f.html.
- Tanja Hagemann and Katerina Katsarou. 2021. A systematic review on anomaly detection for cloud computing environments. In Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference (AICCC '20). Association for Computing Machinery, Kyoto, Japan, 83-96. isbn: 9781450388832. doi: 10.1 145/3442536.3442550.
- Jiajun Jiang et al. 2020. How to mitigate the incident? an efective troubleshooting guide recommendation technique for online service systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020 ).
- Association for Computing Machinery Virtual Event USA 1410-1420. isbn: 9781450370431. doi: 10.1145/3368089.3417054. · DOI · Paper page
- Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Ves Stoyanov and Luke Zettlemoyer. 2019. BART: denoising Sequence-to-Sequence pre-training for natural language generation translation and comprehension (Oct. 2019 ). http://arxiv.org/abs/ 1910.13461 arXiv: 1910. 13461 [cs.CL].
- Patrick Lewis et al. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. Adv. Neural Inf. Process. Syst., 33, 9459-9474. https://p roceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html.
- In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, ( July 2004 ), 74-81. https://aclanthology.org/W04-1013.
- Chang Lou, Cong Chen, Peng Huang, Yingnong Dang, Si Qin, Xinsheng Yang, Xukun Li, Qingwei Lin, and Murali Chintalapati. 2022. { Resin}: a holistic service for dealing with memory leaks in production cloud infrastructure. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 109-125. https://www.usenix.org/conference/osdi22/presentation/lou-resin.
- Chen Luo Jian-Guang Lou Qingwei Lin Qiang Fu Rui Ding Dongmei Zhang and Zhe Wang. 2014. Correlating events with time series for incident diagnosis. · DOI · Paper page
- 10.1145/2623 · doi:10.1145/2623
- Minghua Ma et al. 2020. Diagnosing root causes of intermittent slow queries in cloud databases. Proceedings VLDB Endowment, 13, 8, ( Apr. 2020 ), 1176-1189.
- https://doi.org/10.14778/3389133.3389136. 10.14778/3389133.3389136 · DOI · Paper page
- Aman Madaan et al. 2023. Self-Refine: iterative refinement with Self-Feedback (Mar. 2023 ). http://arxiv.org/abs/2303.17651 arXiv: 2303.17651 [cs.CL]. · arXiv
- Shunyu Yao Jefrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2022. ReAct: synergizing reasoning and acting in language models (Oct. 2022 ). http : / / arxiv. org / abs / 2210. 03629 arXiv: 2210. 03629 [cs.CL].
- Zhengran Zeng et al. 2023. TraceArk: towards actionable performance anomaly alerting for online service systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).
- 10.1109/ICSE-SEIP58684 · doi:10.1109/icse-seip58684
- T Zhang V Kishore F Wu K Q Weinberger et al. 2019. Bertscore: evaluating text generation with bert. arXiv preprint arXiv. https://arxiv.org/abs/ 1904.09675.
Cited by
AI-Augmented SOC: A Survey of LLMs and Agents for Security Automation
Srinivas, S., Kirk, B., Zendejas, J., Espino, M., Boskovich, M., Bari, A., Dajani, K., & Alzahrani, N. (2025) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp5040095 · Cited by 39
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35
“Another line of work explores an agent trying to describe the root cause of a software failure [24].”
Zhang, X., Wang, Q., Li, M., Yuan, Y., Xiao, M., Zhuang, F., & Yu, D. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3629066 · Cited by 15
“RCAcopilot [6] maps multimodal observation data to predefined events and predicts root cause categories in production through an LLM-driven workflow, and agent-based approaches [35] utilize tool-assisted LLMs to actively retrieve diagnostic data like logs and metrics for localization.”
Self-citationLarge Language Models for Security Operations Centers: A Comprehensive Survey
Habibzadeh, A., Feyzi, F., & Atani, R. E. (2026) · Journal of Electrical and Computer Engineering · doi:10.1155/jece/3383674 · Cited by 13
LLexus: an AI agent system for incident management
Las-Casas, P., Kumbhare, A. G., Fonseca, R., & Agarwal, S. (2024) · ACM SIGOPS Operating Systems Review · doi:10.1145/3689051.3689056 · Cited by 14
“However, compared to a fully online approach that analyses and generates a plan at the time of the incident [23, 35], the cost of generating a plan offline and handling incidents using a deterministic execution, as proposed in LLexus , is significantly lower, especially as the number of incidents…”
InfluentialSelf-citation(2025) · doi:10.1145/3737412.3743489 · Cited by 4
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
“Root cause analysis agents (Roy et al., 2024) leverage a multi-agent architecture to solve complex debugging challenges by distributing specialized tasks across different AI components working in tandem.”
Let’s read the log: root cause analysis of railway test execution logs with large language models
Hermawan, R., Bucaioni, A., Enoiu, E., Afzal, W., Saadatmand, M., Zaimovic, N., & Ibtasham, M. S. (2026) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-026-00657-1
Rafi, M. N., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · arXiv · doi:10.48550/arxiv.2412.18750
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
Metadata
- Authors
- Roy, D., Zhang, X., Bhave, R., Bansal, C., Las-Casas, P., Fonseca, R., & Rajmohan, S.
- Venue
- Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 208-219
- OpenAlex
- W4400484653
- Cited by
- 109
- References
- 8
- FWCI
- 49.62
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.