CoChat
Sign Up
Cited by 11Software Engineering ResearchSoftware System Performance and ReliabilitySoftware Engineering Techniques and Practices

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

An Empirical Study on Change-induced Incidents of Online Service Systems

Wu, Yifan, Chai, Bingxu, Li, Ying, Liu, Bingchang, Li, Jianguo, Yang, Yong, Jiang, Wei·2023·2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
DOI 10.1109/icse-seip58684.2023.00027
Open this paper in CoChat — free account

Abstract

Although dedicated efforts have been devoted to ensuring the service quality of online service systems, these systems are still suffering from incidents due to various causes, which lead to user dissatisfaction and economic loss. Change is the most disruptive yet unavoidable maintenance event in online service systems. Among all possible causes of incidents, change is one of the leading causes that induce incidents. To enforce changes with minimized negative impact, change management has been widely applied in industry. However, change-induced incidents are still happening. Most empirical studies involving change-induced incidents are limited to one specific type of incident-inducing change. Moreover, the characteristics of change-induced incidents and challenges of change management have not been studied. To fill the knowledge gap, this paper presents the first empirical study on change-induced incidents of online service systems. 161 real change-induced incidents are collected from a large-scale online service system over two years in Ant Group. By manually examining their post-mortem reports, we clarify the severity of change-induced incidents and analyze the characteristics of change-induced incidents in terms of change types, root causes, and mitigation strategies. Furthermore, we identify a series of vital challenges of change management in practice and point out several practical implications for researchers and engineers. We believe our work could help understand chan…

Overview

Integrity signals

Registry

Crossref · 10.1109/icse-seip58684.2023.00027

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This affects the types of evidence that the agent can safely use.

    “This affects the types of evidence that the agent can safely use [81], [93], [94], [80], [95].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [81]

  • Prompt injection may enter through tickets, runbooks, chat logs, postmortems, or customer-provided text, so retrieved content should remain separate from system instructions, with narrow permissions, provenance records, and independent checks before action.

    “Prompt injection may enter through tickets, runbooks, chat logs, postmortems, or customer-provided text, so retrieved content should remain separate from system instructions, with narrow permissions, provenance records, and independent checks before action [95], [81], [82].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [81]

  • Risks from tool misuse, prompt injection through operational artefacts, drift between verified and realised state, and rollback failure become more serious once the agent is allowed to propose or execute actions.

    “Hallucination or story-mode reasoning can affect every rung, but tool misuse, prompt injection through operational artefacts, drift between verified and realised state, and rollback failure become more serious once the agent is allowed to propose or execute actions [81], [95], [83], [65], [102].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [81]

  • Attackers didn't need to compromise the tool itself, but only needed to influence authorised agents to abuse it.

    “attackers didn't need to compromise the tool itself, but only needed to influence authorised agents to abuse it [99], [81], [150], [42].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [81]

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. 10.1145/2043556.2043572 · DOI
  2. 10.1007/978-3-642-10445-9_18 · DOI
  3. 10.1145/3317550.3321438 · DOI
  4. gunawi, 2016, Why does the cloud stop computing? lessons from hundreds of service outages, Proceedings of the Seventh ACM Symposium on Cloud Computing, 1
  5. 10.1007/978-3-031-14135-5_11 · DOI
  6. 10.1145/3477132.3483577 · DOI
  7. 2021, 2021 Facebook Outage
  8. 10.1145/3368089.3417055 · DOI · Paper page
  9. 2019, Microservices architecture style
  10. zubrow, 2009, IEEE Standard Classification for Software Anomalies
  11. 10.1109/TSC.2016.2539945 · DOI
  12. 10.1145/3447851.3458737 · DOI
  13. 10.1109/DSN.2019.00037 · DOI
  14. mehta, 2020, Rex: Preventing bugs and misconfiguration in large services using correlated change analysis, USENIX Symposium on Networked Systems Design and Implementation (NSDI), 435
  15. 10.1109/ISSRE52982.2021.00017 · DOI
  16. 10.1145/2079296.2079309 · DOI
  17. zhai, 2020, Check before you change: Preventing correlated failures in service updates, USENIX Symposium on Networked Systems Design and Implementation (NSDI), 575
  18. 10.1109/ICSE-SEIP.2019.00020 · DOI
  19. 10.1109/ICSE43902.2021.00085 · DOI
  20. 10.1145/3368089.3417054 · DOI · Paper page
  21. 10.1109/ASE.2019.00042 · DOI
  22. 10.1145/2535372.2535382 · DOI
  23. 2017, Amazon web services s3 outage due to human error
  24. 2016, Google Compute Engine Incident #16007
  25. beyer, 2016, Site Reliability Engineering How Google Runs Production Systems
  26. 10.1145/2716281.2836087 · DOI
  27. li, 2020, Gandalf: An intelligent, end-to-end analytics service for safe deployment in large-scale cloud infrastructure, 17th USENIX Symposium on Networked Systems Design and Implementation ( NSDI 20), 389
  28. 2014, Update on Azure Storage Service Interruption
  29. 10.1145/3468264.3468543 · DOI
  30. 10.1145/3324884.3416624 · DOI
  31. 10.1145/3510457.3513029 · DOI
  32. 10.1109/ICSE-SEIP52600.2021.00043 · DOI · Paper page
  33. 10.1145/2517349.2522727 · DOI
  34. 10.1145/3133888 · DOI
  35. 10.2214/ajr.183.6.01831539 · DOI · Paper page
  36. 10.1109/ISSRE5003.2020.00015 · DOI
  37. 10.1109/INFOCOM.2014.6848128 · DOI
  38. 10.1109/ISSRE52982.2021.00021 · DOI · Paper page
  39. 10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00074 · DOI
  40. sun, 2020, Testing configuration changes in context to prevent production failures, 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 735
  41. xiang, 2020, {PracExtractor}: Extracting configuration good practices from manuals to detect server misconfigurations, 2020 USENIX Annual Technical Conference (USENIX ATC), 265
  42. 0, Google Cloud Status Dashboard
  43. viera, 2005, Understanding interobserver agreement: the kappa statistic, Fam Med, 37, 360
  44. 10.1016/j.future.2019.05.048 · DOI
  45. 0, Azure Status History
  46. 10.1109/TDSC.2004.2 · DOI
  47. 10.1109/32.799955 · DOI
  48. beyer, 2018, The Site Reliability Workbook Practical Ways to Implement SRE
  49. 2021, Increased vm failure rates in a subset of google cloud zones
  50. 10.1145/3452296.3472901 · DOI
  51. 10.1145/2024445.2024454 · DOI

Cited by

Sorted by citations

Metadata

Authors
Wu, Y., Chai, B., Li, Y., Liu, B., Li, J., Yang, Y., & Jiang, W.
Affiliations
  • Peking University (CN)
  • Ant Group (China) (CN)
ORCID
Venue
2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
Publisher
IEEE
Published
2023-05
Pages
234-245
DOI
10.1109/icse-seip58684.2023.00027
OpenAlex
W4383898387
Cited by
11
References
1
Citations by year
2026: 4 · 2025: 2 · 2024: 4 · 2023: 1
FWCI
2.83
Type
conference paper
Language
en
Open access
closed
Licence
Locations
IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
change management, online service systems, incident severity, post-mortem analysis, software maintenance, Ant Group, incident characteristics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review