ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
In large-scale online service systems, the occurrence of software changes is inevitable and frequent. Despite rigorous pre-deployment testing practices, the presence of defective software changes in the online environment cannot be completely eliminated. Consequently, there is a pressing need for automated techniques that can effectively identify these defective changes. However, the current abnormal change detection (ACD) approaches fall short in accurately pinpointing defective changes, primarily due to their disregard for the propagation of faults. To address the limitations of ACD, we propose a novel concept called root cause change analysis (RCCA) to identify the underlying root causes of change-inducing incidents. In order to apply the RCCA concept to practical scenarios, we have devised an intelligent RCCA framework named ChangeRCA. This framework aims to localize the defective change associated with change-inducing incidents among multiple changes. To assess the effectiveness of ChangeRCA, we have conducted an extensive evaluation utilizing a real-world dataset from WeChat and a simulated dataset encompassing 81 diverse defective changes. The evaluation results demonstrate that ChangeRCA outperforms the state-of-the-art ACD approaches, achieving an impressive Top-1 Hit Rate of 85% and significantly reducing the time required to identify defective changes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, causal models have been used to estimate the impact of handover-parameter adjustments and other cellular-network configuration interventions, including transmission-power and cell-offset changes, on subsequent service and performance time series [121], [122], [123].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [121]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Stuart, E. A., Huskamp, H. A., Duckworth, K., Simmons, J., Song, Z., Chernew, M. E., & Barry, C. L. (2014) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-014-0123-z · Cited by 532
The International Arab Journal of Information Technology
(2019) · The International Arab Journal of Information Technology · doi:10.34028/iajit · Cited by 441
Evaluating and Improving Fault Localization
Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., & Keller, B. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.62 · Cited by 426
The Holt-Winters Forecasting Procedure
Chatfield, C. (1978) · Applied Statistics · doi:10.2307/2347162 · Cited by 418
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2887384 · Cited by 410
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Liu, D., Xiang, Q., & He, C. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338961 · Cited by 277
MicroRCA: Root Cause Localization of Performance Issues in Microservices
Wu, L., Tordsson, J., Elmroth, E., & Kao, O. (2020) · NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium · doi:10.1109/noms47738.2020.9110353 · Cited by 265
A general approach to network configuration verification
Beckett, R., Gupta, A., Mahajan, R., & Walker, D. (2017) · Proceedings of the Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3098822.3098834 · Cited by 262
Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-service Environments
Lin, J., Chen, P., & Zheng, Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-03596-9_1 · Cited by 192
Self-citationLocus: locating bugs from software changes
Wen, M., Wu, R., & Cheung, S.-C. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970359 · Cited by 172
Yu, G., Chen, P., Chen, H., Guan, Z., Huang, Z., Jing, L., Weng, T., Sun, X., & Li, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449905 · Cited by 158
Self-citationYu, G., Chen, P., Li, Y., Chen, H., Li, X., & Zheng, Z. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616249 · Cited by 140 · cited 1×
“However, existing RCA approaches [17, 23, 52, 54, 56] primarily rely on observability data, such as metrics and traces, while often neglecting the significance of change data.”
Self-citationBoosting spectrum-based fault localization using PageRank
Zhang, M., Li, X., Zhang, L., & Khurshid, S. (2017) · Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3092703.3092731 · Cited by 137
Continuous deployment at Facebook and OANDA
Savor, T., Douglas, M., Gentili, M., Williams, L., Beck, K., & Stumm, M. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2889223 · Cited by 130
Developing Self-Adaptive Microservice Systems: Challenges and Directions
Mendonça, N. C., Jamshidi, P., Garlan, D., & Pahl, C. (2019) · IEEE Software · doi:10.1109/ms.2019.2955937 · Cited by 101
Microscaler: Automatic Scaling for Microservices with an Online Learning Approach
Yu, G., Chen, P., & Zheng, Z. (2019) · 2019 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2019.00023 · Cited by 97
Self-citationHistorical Spectrum Based Fault Localization
Wen, M., Chen, J., Tian, Y., Wu, R., Hao, D., Han, S., & Cheung, S.-C. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2948158 · Cited by 94
Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings
Wang, H., Wu, Z., Jiang, H., Huang, Y., Wang, J., Kopru, S., & Xie, T. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678708 · Cited by 92
Identifying bad software changes via multimodal anomaly detection for online service systems
Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Xu, H., Zhang, W., Sui, K., & Pei, D. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468543 · Cited by 89
Rapid detection of maintenance induced changes in service performance
Mahimkar, A., Ge, Z., Wang, J., Yates, J., Zhang, Y., Emmons, J., Huntley, B., & Stockert, M. (2011) · Proceedings of the Seventh COnference on emerging Networking EXperiments and Technologies · doi:10.1145/2079296.2079309 · Cited by 88
Practitioners' expectations on automated code comment generation
Hu, X., Xia, X., Lo, D., Wan, Z., Chen, Q., & Zimmermann, T. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510152 · Cited by 85
CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms
Zhang, Y., Guan, Z., Qian, H., Xu, L., Liu, H., Wen, Q., Sun, L., Jiang, J., Fan, L., & Ke, M. (2021) · Proceedings of the 30th ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3481903 · Cited by 84
An Industrial Survey of Safety Evidence Change Impact Analysis Practice
de la Vara, J. L., Borg, M., Wnuk, K., & Moonen, L. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2553032 · Cited by 79
Cotroneo, D., De Simone, L., Liguori, P., Natella, R., & Bidokhti, N. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338916 · Cited by 78
Improving Fault Localization by Integrating Value and Predicate Based Causal Inference Techniques
Kucuk, Y., Henderson, T. A. D., & Podgurski, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00066 · Cited by 76
Detecting the performance impact of upgrades in large operational networks
Mahimkar, A. A., Song, H. H., Ge, Z., Shaikh, A., Wang, J., Yates, J. M., Zhang⋆, Y., & Emmons, J. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851219 · Cited by 72
Actionable and interpretable fault localization for recurring failures in online service systems
Li, Z., Zhao, N., Li, M., Lu, X., Wang, L., Chang, D., Nie, X., Cao, L., Zhang, W., Sui, K., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549092 · Cited by 70
Rapid and robust impact assessment of software changes in large internet-based services
Zhang, S., Liu, Y., Pei, D., Chen, Y., Qu, X., Tao, S., & Zang, Z. (2015) · Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies · doi:10.1145/2716281.2836087 · Cited by 55
How to fight production incidents?: an empirical study on a large-scale cloud service
Ghosh, S., Shetty, M., Bansal, C., & Nath, S. (2022) · Proceedings of the 13th Symposium on Cloud Computing · doi:10.1145/3542929.3563482 · Cited by 54
Understanding and Detecting Software Upgrade Failures in Distributed Systems
Zhang, Y., Yang, J., Jin, Z., Sethi, U., Rodrigues, K., Lu, S., & Yuan, D. (2021) · Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles · doi:10.1145/3477132.3483577 · Cited by 45
Yu, G., Huang, Z., & Chen, P. (2023) · Journal of Software: Evolution and Process · doi:10.1002/smr.2413 · Cited by 44 · cited 2×
“Significant efforts have been made to automate the mitigation of the impact caused by defective changes, with a focus on either abnormal change detection (ACD) [4, 30, 49, 57, 60, 63] or root cause analysis (RCA) [17, 23, 35, 51, 52, 56, 61].”
Self-citationFaster, deeper, easier: crowdsourcing diagnosis of microservice kernel failure from user space
Pan, Y., Ma, M., Jiang, X., & Wang, P. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464805 · Cited by 42 · cited 1×
“Significant efforts have been made to automate the mitigation of the impact caused by defective changes, with a focus on either abnormal change detection (ACD) [4, 30, 49, 57, 60, 63] or root cause analysis (RCA) [17, 23, 35, 51, 52, 56, 61].”
Test-case prioritization for configuration testing
Cheng, R., Zhang, L., Marinov, D., & Xu, T. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464810 · Cited by 38
Graph based Incident Extraction and Diagnosis in Large-Scale Online Systems
He, Z., Chen, P., Luo, Y. B., Yan, Q., Chen, H., Yu, G., & Li, F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556904 · Cited by 30
Self-citationGoing through the Life Cycle of Faults in Clouds: Guidelines on Fault Handling
Li, X., Yu, G., Chen, P., Chen, H., & Chen, Z. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00022 · Cited by 26
Self-citationLi, Y., Yu, G., Chen, P., Zhang, C., & Zheng, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20984-0_15 · Cited by 23
Self-citationLogReducer: Identify and Reduce Log Hotspots in Kernel on the Fly
Yu, G., Chen, P., Li, P., Weng, T., Zheng, H., Deng, Y., & Zheng, Z. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00151 · Cited by 23
Self-citationA composition framework for change management
Mahimkar, A. A., Andrade, C. E., Sinha, R., & Rana, G. (2021) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3452296.3472901 · Cited by 22
Wang, X., Yin, K., Ouyang, Q., Wen, X., Zhang, S., Zhang, W., Cao, L., Han, J., Jin, X., & Pei, D. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00043 · Cited by 20
T-Rank:A Lightweight Spectrum based Fault Localization Approach for Microservice Systems
Ye, Z., Chen, P., & Yu, G. (2021) · 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid) · doi:10.1109/ccgrid51090.2021.00051 · Cited by 19 · cited 1×
“Significant efforts have been made to automate the mitigation of the impact caused by defective changes, with a focus on either abnormal change detection (ACD) [4, 30, 49, 57, 60, 63] or root cause analysis (RCA) [17, 23, 35, 51, 52, 56, 61].”
BackgroundSelf-citationBuildSheriff: change-aware test failure triage for continuous integration builds
Zhang, C., Chen, B., Peng, X., & Zhao, W. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510132 · Cited by 19
An Empirical Study of Crash-inducing Commits in Mozilla Firefox
An, L., & Khomh, F. (2015) · doi:10.1145/2810146.2810152 · Cited by 16
A survey of software architectural change detection and categorization techniques
Mondal, A. K., Schneider, K. A., Roy, B., & Roy, C. K. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111505 · Cited by 14
A system for proactive risk assessment of application changes in cloud operations
Batta, R., Shwartz, L., Nidd, M., Azad, A. P., & Kumar, H. (2021) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud53861.2021.00025 · Cited by 12
Wu, R., Wen, M., Cheung, S.-C., & Zhang, H. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182516 · Cited by 11
Identifying Root-Cause Changes for User-Reported Incidents in Online Service Systems
Zhao, Y., Jiang, L., Tao, Y., Zhang, S., Wu, C., Jia, T., Huang, X., Li, Y., & Wu, Z. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00028 · Cited by 8
Xu, Y. D., Zhang, X., Luo, C. W., Qin, S., Pandey, R., Du, C., Lin, Q., Dang, Y., & Zhou, A. (2021) · doi:10.1145/3447851.3458737 · Cited by 6
NENYA: Cascade Reinforcement Learning for Cost-Aware Failure Mitigation at Microsoft 365
Wang, L., Zhao, P., Du, C., Luo, C. W., Su, M., Yang, F., Liu, Y., Lin, Q., Wang, M., Dang, Y., et al. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539127 · Cited by 4
Research works printed but not linked: 8
- 10.34028/iajit/18/2/14 · DOI
- 10.5555/1656980.1657005 · doi:10.5555/1656980.1657005
- Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Peng Huang, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. 2020. Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. In NSDI 2020. USENIX Association, 389–402. https://www.usenix.org/conference/nsdi20/presentation/li
- Chang Lou, Peng Huang, and Scott Smith. 2020. Understanding, Detecting and Localizing Partial Failures in Large System Software. In NSDI 2020. USENIX Association, 559–574. https://www.usenix.org/conference/nsdi20/presentation/lou
- Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Shekhar Maddila, Balasubramanyan Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2022. Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis. In NSDI 2020. USENIX Association, 435–448. https://www.usenix.org/conference/nsdi20/presentation/mehta
- Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360–363. https://pubmed.ncbi.nlm.nih.gov/15883903/
- Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early Detection of Configuration Errors to Reduce Failure Damage. In OSDI 2016. USENIX Association, 619–634. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/xu
- Ennan Zhai, Ang Chen, Ruzica Piskac, Mahesh Balakrishnan, Bingchuan Tian, Bo Song, and Haoliang Zhang. 2020. Check before You Change: Preventing Correlated Failures in Service Updates. In NSDI 2020. USENIX Association, 575–589. https://www.usenix.org/conference/nsdi20/presentation/zhai
Cited by
Failure Diagnosis in Microservice Systems: A Comprehensive Survey and Analysis
Zhang, S., Xia, S., Fan, W., Shi, B., Xiong, X., Zhong, Z., Ma, M., Sun, Y., & Pei, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715005 · Cited by 57
GALR: Graph-Based Root Cause Localization and LLM-Assisted Recovery for Microservice Systems
Zhang, W., Yang, Z., Peng, F., Zhang, L., Chen, Y., & Chen, R. (2026) · Electronics · doi:10.3390/electronics15010243 · Cited by 7
LLMGuard: Multi-Agent Fault Diagnosis for Reliable Language-Model-as-a-Service
Zhong, Y., Yu, G., Huang, Y., Fu, Q., Ren, R., Feng, C., Yang, Y., & Lyu, M. R. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00018 · Cited by 2
Self-citationLarge language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“…applied without violating checked properties, and whether the system could return quickly to a known-good state if reality objected [88], [102], [103]. d) Pattern D: Closed-loop (self-healing): A closed-loop system aims to integrate detection, diagnosis, mitigation, and recovery verification…”
InfluentialAlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems
Yu, G., Mai, G., Wang, R., Li, R., Chen, P., Pan, L., & Xu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00009 · Cited by 4
Self-citation
Metadata
- Authors
- Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z.
- Affiliations
- Sun Yat-sen University (CN)
- Tencent (China) (CN)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2024-07-12
- Volume
- 1
- Issue
- FSE
- Pages
- 24-46
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4400583494
- Cited by
- 9
- References
- 49
- Citations by year
- 2026: 7 · 2025: 2
- FWCI
- 3.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion)
- Funders
- National Natural Science Foundation of China (No.62272495)
- Guangdong Basic and Applied Basic Research Foundation (No.2023B1515020054)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- fault propagation, online service systems, change localization, WeChat
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.