Identifying bad software changes via multimodal anomaly detection for online service systems
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
In large-scale online service systems, software changes are inevitable and frequent. Due to importing new code or configurations, changes are likely to incur incidents and destroy user experience. Thus it is essential for engineers to identify bad software changes, so as to reduce the influence of incidents and improve system re- liability. To better understand bad software changes, we perform the first empirical study based on large-scale real-world data from a large commercial bank. Our quantitative analyses indicate that about 50.4% of incidents are caused by bad changes, mainly be- cause of code defect, configuration error, resource contention, and software version. Besides, our qualitative analyses show that the current practice of detecting bad software changes performs not well to handle heterogeneous multi-source data involved in soft- ware changes. Based on the findings and motivation obtained from the empirical study, we propose a novel approach named SCWarn aiming to identify bad changes and produce interpretable alerts accurately and timely. The key idea of SCWarn is drawing support from multimodal learning to identify anomalies from heterogeneous multi-source data. An extensive study on two datasets with various bad software changes demonstrates our approach significantly outperforms all the compared approaches, achieving 0.95 F1-score on average and reducing MTTD (mean time to detect) by 20.4%∼60.7%. In particular, we shared some success stories and lessons learned from the practical usage.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network
Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330672 · Cited by 1,804 · cited 7×
“For the compared approaches, we used the parameters provided by their papers and open-source code [39, 56, 64, 70, 77].”
BackgroundMethodInfluentialSelf-citationDetecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., & Soderstrom, T. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219845 · Cited by 1,771 · cited 9×
“Compared with traditional Recurrent Neural Network (RNN), LSTM has shown its strong capability to maintain the memory of long-term dependencies due to a contextbasedweighted self-loop that allows them to forget past information in addition to accumulating it [32].”
InfluentialDeep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 4×
“To tackle these challenges, we leverage the idea of multimodal learning to identify anomalies in an unsupervised manner, which has been widely used to learn information from cross modality (e.g., text and images) [59].”
InfluentialDrain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · cited 3×
“We adopt the state-of-the-art log parsing algorithm, Drain [25], to extract log templates, and its superiority and efficiency have been demonstrated in [83].”
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
Self-citationLogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs
Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P., et al. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/658 · Cited by 708
Self-citationTime-Series Anomaly Detection Service at Microsoft
Ren, H., Xu, B., Wang, Y., Yi, C., Huang, C., Kou, X., Xing, T., Yang, M., Tong, J., & Zhang, Q. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330680 · Cited by 623
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574 · cited 1×
“We adopt the state-of-the-art log parsing algorithm, Drain [25], to extract log templates, and its superiority and efficiency have been demonstrated in [83].”
Cited by
Yu, G., Chen, P., Li, Y., Chen, H., Li, X., & Zheng, Z. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616249 · Cited by 140
Heterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention
Lee, C., Yang, T., Chen, Z., Su, Y., Yang, Y., & Lyu, M. R. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00148 · Cited by 68
Twin Graph-Based Anomaly Detection via Attentive Multi-Modal Learning for Microservice System
Huang, J., Yang, Y., Yu, H., Li, J., & Zheng, X. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00138 · Cited by 50
“Framework of the proposed method MSTGAD multi-modal data, such as DeepTraLog [1], SCWarn [23], and HADES [24].”
BackgroundMethodInfluentialSelf-citationPUTraceAD: Trace Anomaly Detection with Partial Labels based on GNN and PU Learning
Zhang, K., Zhang, C., Peng, X., & Sha, C. (2022) · 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre55969.2022.00032 · Cited by 33
“A. Experimental Design 1) Dataset: Our experiments are conducted on TrainTicket [25], [26], a microservice benchmark system widely used in researches on microservice architecture, anomaly detection, and root cause analysis [5], [7], [27].”
MethodMonitorAssistant: simplifying cloud service monitoring via large language models
Yu, Z., Ma, M., Zhang, C., Qin, S., Kang, Y., Bansal, C., Rajmohan, S., Dang, Y., Pei, C., Pei, D., et al. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663826 · Cited by 32
Self-citation
Metadata
- Authors
- Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Xu, H., Zhang, W., Sui, K., & Pei, D.
- Venue
- Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2021-08-18
- Pages
- 527-539
- OpenAlex
- W3194768773
- Cited by
- 89
- References
- 15
- FWCI
- 12.47
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying bad software changes via multimodal anomaly detection for online service systems. https://app.cochat.ai/papers/10.1145/3468264.3468543
- The paper
- Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Xu, H., Zhang, W., Sui, K., & Pei, D. (2021). Identifying bad software changes via multimodal anomaly detection for online service systems. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. https://doi.org/10.1145/3468264.3468543
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.