How to mitigate the incident? An effective troubleshooting guide recommendation technique for online service systems
Abstract
In recent years, more and more traditional shrink-wrapped software is provided as 7x24 online services. Incidents (events that lead to service disruptions or outages) could affect service availability and cause great financial loss. Therefore, mitigating the incidents is important and time critical. In practice, a document describing a mitigation process, called a troubleshooting guide (TSG), is usually used to reduce the Time To Mitigate (TTM). To investigate the usage of TSGs in real-world online services, we conduct the first empirical study on 18 real-world, large-scale online service systems in Microsoft. We analyze the distribution and characteristics of TSGs among all incident records in the past two years. According to our study, 27.2% incidents have TSG records and 36.2% of them occurred at least twice. Besides, on average developers spend around 36.3% of the entire mitigation time on locating the desired TSGs.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Queries, dashboards, validators, runbooks, ticketing systems, change windows, approvals, and audit trails still define what can be safely done.
“Queries, dashboards, validators, runbooks, ticketing systems, change windows, approvals, and audit trails still define what can be safely done [4], [5], [72], [70], [73], [55].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [72]
Once an agent proposes mitigations, the system becomes a socio-technical control loop, and the failure mode is no longer wrong answer but wrong change at the wrong time.
“Once an agent proposes mitigations (restart, failover, throttling, config toggle), the system becomes a socio-technical control loop.The failure mode is no longer wrong answer but wrong change at the wrong time [4], [72], [105], [85].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [72]
Trusted systems leave traces that can be reviewed, act with care during uncertainty, and recover when things go off plan.
“trusted systems leave traces that can be reviewed, act with care during uncertainty, and recover when things go off plan [72], [11], [105], [4].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [72]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169 · see citing works
Zhou, J., Zhang, H., & Lo, D. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227210 · Cited by 507 · see citing works
Self-citationAdaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability
Yang, Y., Zhou, D.-W., Zhan, D.-C., Xiong, H., & Jiang, Y. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330865 · Cited by 83 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., et al. (2024) · Proceedings of the Nineteenth European Conference on Computer Systems · doi:10.1145/3627703.3629553 · Cited by 190 · see citing works
Self-citationAssess and Summarize: Improve Outage Understanding with Large Language Models
Jin, P., Zhang, S., Ma, M., Li, H., Kang, Y., Li, L., Liu, Y., Qiao, B., Zhang, C., Zhao, P., et al. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613891 · Cited by 50 · see citing works
Self-citationAI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges
Cheng, Q., Sahoo, D., Saha, A., Yang, W., Liu, C., Woo, G., Singh, M., Saverese, S., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2304.04661 · Cited by 23 · see citing works
LogExpert: Log-based Recommended Resolutions Generation using Large Language Model
Wang, J., Chu, G., Wang, J., Sun, H., Qi, Q., Wang, Y., Qi, J., & Liao, J. (2024) · Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results · doi:10.1145/3639476.3639773 · Cited by 12 · see citing works
AIOpsLab in Action: An Open Platform for AIOps Research
Ma, M., Clark, J., & Zhang, S. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3728619
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075
- Authors
- Jiang, J., Lu, W., Chen, J., Lin, Q., Zhao, P., Kang, Y., Zhang, H., Xiong, Y., Gao, F., Xu, Z., Dang, Y., & Zhang, D.
- Affiliations
- Tianjin University (CN)
- Peking University (CN)
- Microsoft Research Asia (China) (CN)
- University of Newcastle Australia (AU)
- Microsoft (United States) (US)
- ORCID
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1410-1420
- OpenAlex
- W3086623952
- Cited by
- 56
- References
- 6
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 17 · 2023: 5 · 2022: 11 · 2021: 7 · 2020: 3
- FWCI
- 5.45
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- NOVA (University of Newcastle Australia) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- troubleshooting guide, incident mitigation, online service systems, service availability, outages, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
