CoChat

Automatic Root Cause Analysis via Large Language Models for Cloud Incidents

Open access
Chen, Yinfang, Xie, Huaibing, Ma, Minghua, Kang, Yu, Gao, Xin, Shi, Liu, Cao, Yunjie, Gao, Xuedong, Fan, Hao, Wen, Ming, Zeng, Jun, Ghosh, Supriyo, Zhang, Xuchao, Zhang, Chaoyun, Lin, Qingwei, Rajmohan, Saravan, Zhang, Dongmei, Xu, Tianyin · 2024 · Proceedings of the Nineteenth European Conference on Computer Systems

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.

Abstract

Ensuring the reliability and availability of cloud services necessitates efficient root cause analysis (RCA) for cloud incidents. Traditional RCA methods, which rely on manual investigations of data sources such as logs and traces, are often laborious, error-prone, and challenging for on-call engineers. In this paper, we introduce RCACopilot, an innovative on-call system empowered by the large language model for automating RCA of cloud incidents. RCACopilot matches incoming incidents to corresponding incident handlers based on their alert types, aggregates the critical runtime diagnostic information, predicts the incident's root cause category, and provides an explanatory narrative. We evaluate RCACopilot using a real-world dataset consisting of a year's worth of incidents from Microsoft. Our evaluation demonstrates that RCACopilot achieves RCA accuracy up to 0.766. Furthermore, the diagnostic information collection component of RCACopilot has been successfully in use at Microsoft for over four years.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3627703.3629553

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 2 Recurring incidents proportion vs. time interval.
Figure 2 Recurring incidents proportion vs. time interval.CC-BY · From the PDF
Figure 3 Distribution of incident category frequency.
Figure 3 Distribution of incident category frequency.CC-BY · From the PDF
Figure 4 RCACopilot architecture.
Figure 4 RCACopilot architecture.CC-BY · From the PDF
Figure 5 A RCACopilot handler for too many messages stuck in the delivery queue alert.
Figure 5 A RCACopilot handler for too many messages stuck in the delivery queue alert.CC-BY · From the PDF
Figure 9 The prompt to predict incident category.
Figure 9 The prompt to predict incident category.CC-BY · From the PDF
Figure 10 Web-based user interface of RCACopilot for handler construction.
Figure 10 Web-based user interface of RCACopilot for handler construction.CC-BY · From the PDF
Figure 12 Effectiveness of using different K and alpha.
Figure 12 Effectiveness of using different K and alpha.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

12 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 55 references

Research works printed but not linked: 9

  1. Chen, Y., Sun, X., Nath, S., Yang, Z., and Xu, T. Push-Button Reliability Testing for Cloud-Backed Applications with Rainmaker. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI'23) (2023).
  2. Lou, C., Chen, C., Huang, P., Dang, Y., Qin, S., Yang, X., Li, X., Lin, Q., and Chintalapati, M. RESIN: A holistic service for dealing with memory leaks in production cloud infrastructure. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22) (2022).
  3. Lou, C., Huang, P., and Smith, S. Understanding, detecting and localizing partial failures in large system software. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI'20) (2020).
  4. Ma, M., Zhang, S., Chen, J., Xu, J., Li, H., Lin, Y., Nie, X., Zhou, B., Wang, Y., and Pei, D. Jump-starting multivariate time series anomaly detection for online service systems. In 2021 USENIX Annual Technical Conference (ATC'21) (2021).
  5. Sun, X., Cheng, R., Chen, J., Ang, E., Legunsen, O., and Xu, T. Testing Configuration Changes in Context to Prevent Production Failures. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI'20) (2020).
  6. Sun, X., Luo, W., Gu, J. T., Ganesan, A., Alagappan, R., Gasch, M., Suresh, L., and Xu, T. Automatic Reliability Testing for Cluster Management Controllers. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22) (2022).
  7. Wu, Y., Chen, A., Haeberlen, A., Zhou, W., and Loo, B. T. Automated bug removal for software-defined networks. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI'17) (2017).
  8. Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Zhao, X., Zhang, Y., Jain, P., and Stumm, M. Simple testing can prevent most critical failures: An analysis of production failures in distributed data-intensive systems. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI'14) (2014).
  9. 10.1109/icse-seip58684.2023.00029 · DOI · Paper page

Cited by

See all 442

Metadata

Authors
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., Zeng, J., Ghosh, S., Zhang, X., Zhang, C., Lin, Q., Rajmohan, S., Zhang, D., & Xu, T.
Affiliations
  • Huazhong University of Science and Technology (CN)
  • Peking University (CN)
  • National University of Singapore (SG)
  • Microsoft (United States) (US)
  • University of Illinois Urbana-Champaign (US)
Venue
Proceedings of the Nineteenth European Conference on Computer Systems
Publisher
ACM
Published
2024-04-22
Pages
674-688
OpenAlex
W4394946189
Cited by
190
References
48
Citations by year
2026: 89 · 2025: 59 · 2024: 41
FWCI
82.63
Type
conference paper
Language
en
Open access
Locations
  • European Conference on Computer Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
root cause analysis, large language models, incident diagnosis, cloud reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. https://app.cochat.ai/papers/10.1145/3627703.3629553
The paper
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., Zeng, J., Ghosh, S., Zhang, X., Zhang, C., Lin, Q., Rajmohan, S., Zhang, D., & Xu, T. (2024). Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. Proceedings of the Nineteenth European Conference on Computer Systems. https://doi.org/10.1145/3627703.3629553

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.