ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
In large-scale online service systems, the occurrence of software changes is inevitable and frequent. Despite rigorous pre-deployment testing practices, the presence of defective software changes in the online environment cannot be completely eliminated. Consequently, there is a pressing need for automated techniques that can effectively identify these defective changes. However, the current abnormal change detection (ACD) approaches fall short in accurately pinpointing defective changes, primarily due to their disregard for the propagation of faults. To address the limitations of ACD, we propose a novel concept called root cause change analysis (RCCA) to identify the underlying root causes of change-inducing incidents. In order to apply the RCCA concept to practical scenarios, we have devised an intelligent RCCA framework named ChangeRCA. This framework aims to localize the defective change associated with change-inducing incidents among multiple changes. To assess the effectiveness of ChangeRCA, we have conducted an extensive evaluation utilizing a real-world dataset from WeChat and a simulated dataset encompassing 81 diverse defective changes. The evaluation results demonstrate that ChangeRCA outperforms the state-of-the-art ACD approaches, achieving an impressive Top-1 Hit Rate of 85% and significantly reducing the time required to identify defective changes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, causal models have been used to estimate the impact of handover-parameter adjustments and other cellular-network configuration interventions, including transmission-power and cell-offset changes, on subsequent service and performance time series [121], [122], [123].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [121]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Stuart, E. A., Huskamp, H. A., Duckworth, K., Simmons, J., Song, Z., Chernew, M. E., & Barry, C. L. (2014) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-014-0123-z · Cited by 532
The International Arab Journal of Information Technology
(2019) · The International Arab Journal of Information Technology · doi:10.34028/iajit · Cited by 441
Evaluating and Improving Fault Localization
Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., & Keller, B. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.62 · Cited by 426
The Holt-Winters Forecasting Procedure
Chatfield, C. (1978) · Applied Statistics · doi:10.2307/2347162 · Cited by 418
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2887384 · Cited by 410
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Liu, D., Xiang, Q., & He, C. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338961 · Cited by 277
MicroRCA: Root Cause Localization of Performance Issues in Microservices
Wu, L., Tordsson, J., Elmroth, E., & Kao, O. (2020) · NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium · doi:10.1109/noms47738.2020.9110353 · Cited by 265
A general approach to network configuration verification
Beckett, R., Gupta, A., Mahajan, R., & Walker, D. (2017) · Proceedings of the Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3098822.3098834 · Cited by 262
Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-service Environments
Lin, J., Chen, P., & Zheng, Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-03596-9_1 · Cited by 192
Self-citation
Research works printed but not linked: 8
- 10.34028/iajit/18/2/14 · DOI
- 10.5555/1656980.1657005 · doi:10.5555/1656980.1657005
- Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Peng Huang, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. 2020. Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. In NSDI 2020. USENIX Association, 389–402. https://www.usenix.org/conference/nsdi20/presentation/li
- Chang Lou, Peng Huang, and Scott Smith. 2020. Understanding, Detecting and Localizing Partial Failures in Large System Software. In NSDI 2020. USENIX Association, 559–574. https://www.usenix.org/conference/nsdi20/presentation/lou
- Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Shekhar Maddila, Balasubramanyan Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2022. Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis. In NSDI 2020. USENIX Association, 435–448. https://www.usenix.org/conference/nsdi20/presentation/mehta
- Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360–363. https://pubmed.ncbi.nlm.nih.gov/15883903/
- Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early Detection of Configuration Errors to Reduce Failure Damage. In OSDI 2016. USENIX Association, 619–634. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/xu
- Ennan Zhai, Ang Chen, Ruzica Piskac, Mahesh Balakrishnan, Bingchuan Tian, Bo Song, and Haoliang Zhang. 2020. Check before You Change: Preventing Correlated Failures in Service Updates. In NSDI 2020. USENIX Association, 575–589. https://www.usenix.org/conference/nsdi20/presentation/zhai
Cited by
Failure Diagnosis in Microservice Systems: A Comprehensive Survey and Analysis
Zhang, S., Xia, S., Fan, W., Shi, B., Xiong, X., Zhong, Z., Ma, M., Sun, Y., & Pei, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715005 · Cited by 57
GALR: Graph-Based Root Cause Localization and LLM-Assisted Recovery for Microservice Systems
Zhang, W., Yang, Z., Peng, F., Zhang, L., Chen, Y., & Chen, R. (2026) · Electronics · doi:10.3390/electronics15010243 · Cited by 7
LLMGuard: Multi-Agent Fault Diagnosis for Reliable Language-Model-as-a-Service
Zhong, Y., Yu, G., Huang, Y., Fu, Q., Ren, R., Feng, C., Yang, Y., & Lyu, M. R. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00018 · Cited by 2
Self-citationLarge language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“…applied without violating checked properties, and whether the system could return quickly to a known-good state if reality objected [88], [102], [103]. d) Pattern D: Closed-loop (self-healing): A closed-loop system aims to integrate detection, diagnosis, mitigation, and recovery verification…”
InfluentialAlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems
Yu, G., Mai, G., Wang, R., Li, R., Chen, P., Pan, L., & Xu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00009 · Cited by 4
Self-citation
Metadata
- Authors
- Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z.
- Affiliations
- Sun Yat-sen University (CN)
- Tencent (China) (CN)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2024-07-12
- Volume
- 1
- Issue
- FSE
- Pages
- 24-46
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4400583494
- Cited by
- 9
- References
- 49
- Citations by year
- 2026: 7 · 2025: 2
- FWCI
- 3.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion)
- Funders
- National Natural Science Foundation of China (No.62272495)
- Guangdong Basic and Applied Basic Research Foundation (No.2023B1515020054)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault propagation, online service systems, change localization, WeChat
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems. https://app.cochat.ai/papers/10.1145/3643728
- The paper
- Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z. (2024). ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems. Proceedings of the ACM on Software Engineering. https://doi.org/10.1145/3643728
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.