CoChat

Exploring LLM-Based Agents for Root Cause Analysis

Open access
Roy, Devjeet, Zhang, Xuchao, Bhave, Rashi, Bansal, Chetan, Las-Casas, Pedro, Fonseca, Rodrigo, Rajmohan, Saravan · 2024 · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering
Cited by 109

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team’s specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized Large Language Models (LLMs) to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at a large IT corporation. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3663529.3663841

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 21

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, Ann Arbor, Michigan, ( June 2005 ), 65-72. https://aclanthology.org/W05-0909.
  2. Chetan Bansal, Sundararajan Renganathan, Ashima Asudani, Olivier Midy, and Mathru Janakiraman. 2019. Decaf: diagnosing and triaging performance issues in large-scale cloud services. CoRR, abs/ 1910.05339. http://arxiv.org/abs / 1910.05339 arXiv: 1910.05339. · arXiv
  3. Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304. 05128. http://ar xiv. org/abs/2304.05128.
  4. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 1701-1716. https://www.usenix.org/conference/nsdi23/presentation /chen-yinfang.
  5. Proceedings of Machine Learning Research 202 10764-10799. Andreas Krause Emma Brunskill Kyunghyun Cho Barbara Engelhardt Sivan Sabato and Jonathan Scarlett (Eds.) https://proceedings.mlr.press/v202/gao23f.html.
  6. Tanja Hagemann and Katerina Katsarou. 2021. A systematic review on anomaly detection for cloud computing environments. In Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference (AICCC '20). Association for Computing Machinery, Kyoto, Japan, 83-96. isbn: 9781450388832. doi: 10.1 145/3442536.3442550.
  7. Jiajun Jiang et al. 2020. How to mitigate the incident? an efective troubleshooting guide recommendation technique for online service systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020 ).
  8. Association for Computing Machinery Virtual Event USA 1410-1420. isbn: 9781450370431. doi: 10.1145/3368089.3417054. · DOI · Paper page
  9. Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Ves Stoyanov and Luke Zettlemoyer. 2019. BART: denoising Sequence-to-Sequence pre-training for natural language generation translation and comprehension (Oct. 2019 ). http://arxiv.org/abs/ 1910.13461 arXiv: 1910. 13461 [cs.CL].
  10. Patrick Lewis et al. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. Adv. Neural Inf. Process. Syst., 33, 9459-9474. https://p roceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html.
  11. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, ( July 2004 ), 74-81. https://aclanthology.org/W04-1013.
  12. Chang Lou, Cong Chen, Peng Huang, Yingnong Dang, Si Qin, Xinsheng Yang, Xukun Li, Qingwei Lin, and Murali Chintalapati. 2022. { Resin}: a holistic service for dealing with memory leaks in production cloud infrastructure. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 109-125. https://www.usenix.org/conference/osdi22/presentation/lou-resin.
  13. Chen Luo Jian-Guang Lou Qingwei Lin Qiang Fu Rui Ding Dongmei Zhang and Zhe Wang. 2014. Correlating events with time series for incident diagnosis. · DOI · Paper page
  14. 10.1145/2623 · doi:10.1145/2623
  15. Minghua Ma et al. 2020. Diagnosing root causes of intermittent slow queries in cloud databases. Proceedings VLDB Endowment, 13, 8, ( Apr. 2020 ), 1176-1189.
  16. https://doi.org/10.14778/3389133.3389136. 10.14778/3389133.3389136 · DOI · Paper page
  17. Aman Madaan et al. 2023. Self-Refine: iterative refinement with Self-Feedback (Mar. 2023 ). http://arxiv.org/abs/2303.17651 arXiv: 2303.17651 [cs.CL]. · arXiv
  18. Shunyu Yao Jefrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2022. ReAct: synergizing reasoning and acting in language models (Oct. 2022 ). http : / / arxiv. org / abs / 2210. 03629 arXiv: 2210. 03629 [cs.CL].
  19. Zhengran Zeng et al. 2023. TraceArk: towards actionable performance anomaly alerting for online service systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).
  20. 10.1109/ICSE-SEIP58684 · doi:10.1109/icse-seip58684
  21. T Zhang V Kishore F Wu K Q Weinberger et al. 2019. Bertscore: evaluating text generation with bert. arXiv preprint arXiv. https://arxiv.org/abs/ 1904.09675.

Cited by

See all 109

Metadata

Authors
Roy, D., Zhang, X., Bhave, R., Bansal, C., Las-Casas, P., Fonseca, R., & Rajmohan, S.
Venue
Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering
Publisher
ACM
Published
2024-07-10
Pages
208-219
OpenAlex
W4400484653
Cited by
109
References
8
FWCI
49.62
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Exploring LLM-Based Agents for Root Cause Analysis. https://app.cochat.ai/papers/10.1145/3663529.3663841
The paper
Roy, D., Zhang, X., Bhave, R., Bansal, C., Las-Casas, P., Fonseca, R., & Rajmohan, S. (2024). Exploring LLM-Based Agents for Root Cause Analysis. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. https://doi.org/10.1145/3663529.3663841

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.