How incidental are the incidents?: characterizing and prioritizing incidents for large-scale online service systems
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Although tremendous efforts have been devoted to the quality assurance of online service systems, in reality, these systems still come across many incidents (i.e., unplanned interruptions and outages), which can decrease user satisfaction or cause economic loss. To better understand the characteristics of incidents and improve the incident management process, we perform the first large-scale empirical analysis of incidents collected from 18 real-world online service systems in Microsoft. Surprisingly, we find that although a large number of incidents could occur over a short period of time, many of them actually do not matter, i.e., engineers will not fix them with a high priority after manually identifying their root cause. We call these incidents incidental incidents. Our qualitative and quantitative analyses show that incidental incidents are significant in terms of both number and cost. Therefore, it is important to prioritize incidents by identifying incidental incidents in advance to optimize incident management efforts. In particular, we propose an approach, called DeepIP (Deep learning based Incident Prioritization), to prioritizing incidents based on a large amount of historical incident data. More specifically, we design an attention-based Convolutional Neural Network (CNN) to learn a prediction model to identify incidental incidents. We then prioritize all incidents by ranking the predicted probabilities of incidents being incidental. We evaluate the performance of DeepIP using real-world incident data. The experimental results show that DeepIP effectively prioritizes incidents by identifying incidental incidents and significantly outperforms all the compared approaches. For example, the AUC of DeepIP achieves 0.808, while that of the best compared approach is only 0.624 on average.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835 · cited 2×
“Here, we draw support from deep learning, since the features we used are mainly textual information, and deep learning can achieve semantic understanding of natural-language descriptions and has been demonstrated to outperform traditional machine learning algorithms [10, 16].”
BackgroundMethodFastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887 · cited 1×
“DeepIP then embeds each word in the textual description of an incident report into a vector by using a word vector, which is pre-trained on historical incident data using the FastText algorithm [28].”
MethodRobust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
Self-citationUnderstanding network failures in data centers: measurement, analysis, and implications
Gill, P., Jain, N., & Nagappan, N. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018477 · Cited by 714 · cited 1×
“Over the years, there have been many empirical studies on the characteristics of failures of data centers and cloud systems [20, 42, 45, 56].”
BackgroundApples-to-apples in cross-validation studies: pitfalls in classifier performance measurement
Forman, G., & Scholz, M. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1882471.1882479 · Cited by 368 · cited 1×
“In particular, as demonstrated in the existing work [19], we calculate the mean of all the classes for the two metrics.”
MethodRoot-Cause Metric Location for Microservice Systems via Log Anomaly Detection
Wang, L., Zhao, N., Chen, J., Li, P., Zhang, W., & Sui, K. (2020) · 2020 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws49710.2020.00026 · Cited by 84
Self-citationJiang, J., Lu, W., Chen, J., Lin, Q., Zhao, P., Kang, Y., Zhang, H., Xiong, Y., Gao, F., Xu, Z., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417054 · Cited by 56
Self-citationIdentifying linked incidents in large-scale online service systems
Chen, Y., Yang, X., Dong, H., He, X., Zhang, H., Lin, Q., Chen, J., Zhao, P., Kang, Y., Gao, F., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409768 · Cited by 47
Self-citation
Cited by
Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., et al. (2024) · Proceedings of the Nineteenth European Conference on Computer Systems · doi:10.1145/3627703.3629553 · Cited by 190
Self-citationDeep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundSelf-citationRecommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
An empirical investigation of practical log anomaly detection for online service systems
Zhao, N., Wang, H., Li, Z., Peng, X., Wang, G., Pan, Z., Wu, Y., Feng, Z., Wen, X., Zhang, W., et al. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3473933 · Cited by 77
Muffin: testing deep learning libraries via neural architecture fuzzing
Gu, J., Luo, X., Zhou, Y., & Wang, X. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510092 · Cited by 71
“DL systems ( i.e. , software systems based on DL techniques) have been widely adopted in various domains in practice, e.g. , self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
MethodJiang, J., Lu, W., Chen, J., Lin, Q., Zhao, P., Kang, Y., Zhang, H., Xiong, Y., Gao, F., Xu, Z., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417054 · Cited by 56
Self-citationAssess and Summarize: Improve Outage Understanding with Large Language Models
Jin, P., Zhang, S., Ma, M., Li, H., Kang, Y., Li, L., Liu, Y., Qiao, B., Zhang, C., Zhao, P., et al. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613891 · Cited by 50
Self-citationMonitorAssistant: simplifying cloud service monitoring via large language models
Yu, Z., Ma, M., Zhang, C., Qin, S., Kang, Y., Bansal, C., Rajmohan, S., Dang, Y., Pei, C., Pei, D., et al. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663826 · Cited by 32
Self-citationAn empirical study on change-induced incidents of online service systems
Wu, Y., Chai, B., Li, Y., Liu, B., Li, J., Yang, Y., & Jiang, W. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00027 · Cited by 11
AIOpsLab in Action: An Open Platform for AIOps Research
Ma, M., Clark, J., & Zhang, S. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3728619
Self-citation
Metadata
- Authors
- Chen, J., Zhang, S., He, X., Lin, Q., Zhang, H., Hao, D., Kang, Y., Gao, F., Xu, Z., Dang, Y., & Zhang, D.
- Venue
- Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2020-12-21
- Pages
- 373-384
- OpenAlex
- W3047008933
- Cited by
- 37
- References
- 9
- FWCI
- 4.60
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.