CoChat

ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems

Yu, Guangba, Chen, Pengfei, He, Zilong, Yan, Qiuyu, Luo, Yu, Li, Fangyuan, Zheng, Zibin · 2024 · Proceedings of the ACM on Software Engineering

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

In large-scale online service systems, the occurrence of software changes is inevitable and frequent. Despite rigorous pre-deployment testing practices, the presence of defective software changes in the online environment cannot be completely eliminated. Consequently, there is a pressing need for automated techniques that can effectively identify these defective changes. However, the current abnormal change detection (ACD) approaches fall short in accurately pinpointing defective changes, primarily due to their disregard for the propagation of faults. To address the limitations of ACD, we propose a novel concept called root cause change analysis (RCCA) to identify the underlying root causes of change-inducing incidents. In order to apply the RCCA concept to practical scenarios, we have devised an intelligent RCCA framework named ChangeRCA. This framework aims to localize the defective change associated with change-inducing incidents among multiple changes. To assess the effectiveness of ChangeRCA, we have conducted an extensive evaluation utilizing a real-world dataset from WeChat and a simulated dataset encompassing 81 diverse defective changes. The evaluation results demonstrate that ChangeRCA outperforms the state-of-the-art ACD approaches, achieving an impressive Top-1 Hit Rate of 85% and significantly reducing the time required to identify defective changes.

Overview

Integrity signals

Registry

Crossref · 10.1145/3643728

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 49 references

Research works printed but not linked: 8

  1. 10.34028/iajit/18/2/14 · DOI
  2. 10.5555/1656980.1657005 · doi:10.5555/1656980.1657005
  3. Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Peng Huang, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. 2020. Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. In NSDI 2020. USENIX Association, 389–402. https://www.usenix.org/conference/nsdi20/presentation/li
  4. Chang Lou, Peng Huang, and Scott Smith. 2020. Understanding, Detecting and Localizing Partial Failures in Large System Software. In NSDI 2020. USENIX Association, 559–574. https://www.usenix.org/conference/nsdi20/presentation/lou
  5. Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Shekhar Maddila, Balasubramanyan Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2022. Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis. In NSDI 2020. USENIX Association, 435–448. https://www.usenix.org/conference/nsdi20/presentation/mehta
  6. Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360–363. https://pubmed.ncbi.nlm.nih.gov/15883903/
  7. Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early Detection of Configuration Errors to Reduce Failure Damage. In OSDI 2016. USENIX Association, 619–634. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/xu
  8. Ennan Zhai, Ang Chen, Ruzica Piskac, Mahesh Balakrishnan, Bingchuan Tian, Bo Song, and Haoliang Zhang. 2020. Check before You Change: Preventing Correlated Failures in Service Updates. In NSDI 2020. USENIX Association, 575–589. https://www.usenix.org/conference/nsdi20/presentation/zhai

Cited by

See all 10

Metadata

Authors
Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z.
Affiliations
  • Sun Yat-sen University (CN)
  • Tencent (China) (CN)
Venue
Proceedings of the ACM on Software Engineering
Published
2024-07-12
Volume
1
Issue
FSE
Pages
24-46
ISSN
2994-970X (ISSN-L)
OpenAlex
W4400583494
Cited by
9
References
49
Citations by year
2026: 7 · 2025: 2
FWCI
3.39
Type
journal article
Language
en
Open access
closed
Locations
Proceedings of the ACM on software engineering. (publishedVersion)
Funders
  • National Natural Science Foundation of China (No.62272495)
  • Guangdong Basic and Applied Basic Research Foundation (No.2023B1515020054)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
fault propagation, online service systems, change localization, WeChat

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems. https://app.cochat.ai/papers/10.1145/3643728
The paper
Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z. (2024). ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems. Proceedings of the ACM on Software Engineering. https://doi.org/10.1145/3643728

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.