CoChat
Sign Up
Cited by 14Software System Performance and ReliabilitySoftware Engineering Techniques and PracticesDistributed systems and fault tolerance

Failures and fixes: a study of software system incident response

Open access
Sillito, Jonathan, Kutomi, Esdras·2020·2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)
DOI 10.1109/icsme46990.2020.00027
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper presents the results of a research study related to software system failures, with the goal of understanding how we might better evolve, maintain and support software systems in production. We have qualitatively analyzed thirty incidents: fifteen collected through in depth interviews with engineers, and fifteen sampled from publicly published incident reports (generally produced as part of postmortem reviews). Our analysis focused on understanding and categorizing how failures occurred, and how they were detected, investigated and mitigated. We also captured analytic insights related to the current state of the practice and associated challenges in the form of 11 key observations. For example, we observed that failures can cascade through a system leading to major outages; and that often engineers do not understand the scaling limits of systems they are supporting until those limits are exceeded. We argue that the challenges we have identified can lead to improvements to how systems are engineered and supported.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icsme46990.2020.00027

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In guarded planner-executor systems, the human reviews a bundle that can be scored, archived, and replayed, not a chat transcript.

    “In guarded planner-executor systems, the human does not review a chat transcript.They review a bundle that can be scored, archived, and replayed [85], [70], [102], [74].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [102]

    The Study Setup describes an analysis of software incidents, and the Investigating section describes responders using dashboards, logs, and other diagnostic information. Neither the supplied passages nor searches for planner-executor review and scored, archived, or replayed bundles address the claimed human-review process. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Rob Barrett, Eser Kandogan, Paul P. Maglio, Eben M. Haber, Leila A. Takayama, and Madhu Prebaker. Field studies of computer system administrators: Analysis of system manageemnt tools and practices. In Proceedings of the Conference on Computer-Supported Cooperative Work (CSCW), pages 388-395, 2004.
  2. Ivan Beschastnikh, Yuriy Brun, Michael D. Ernst, and Arvind Kr- ishnamurthy. Inferring models of concurrent systems from logs of their behavior with csight. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, page 468479, New York, NY, USA, 2014. Association for Computing Machinery.
  3. Ivan Beschastnikh, Patty Wang, Yuriy Brun, and Michael D. Ernst. Debugging distributed systems. Commun. ACM, 59(8):3237, July 2016.
  4. Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media, USA, 2016.
  5. Finn Olav Bjrnson, Alf Inge Wang, and Erik Arisholm. Improving the effectiveness of root cause analysis in post mortem analysis: A controlled experiment. Information and Software Technology, 51(1):150-161, 2009. Special Section -Most Cited Articles in 2002 and Regular Research Papers.
  6. Peter Bodik, Greg Friedman, Lukas Biewald, Helen Levine, George Candea, Kayur Patel, Gilman Tolle, Jon Hui, Armando Fox, Michael I. Jordan, and David Patterson. Combining visualization and statistical analysis to improve operator confidence and efficiency for failure detec- tion and localization. In Second International Conference on Autonomic Computing (ICAC'05), pages 89-100, 2005.
  7. Davide G. Cavezza, Roberto Pietrantuono, Javier Alonso, Stefano Russo, and Kishor S. Trivedi. Reproducibility of environment-dependent software failures: An experience report. In 2014 IEEE 25th International Symposium on Software Reliability Engineering, pages 267-276, 2014.
  8. Edmund M. Clarke, E. Allen Emerson, and Aravinda P. Sistla. Automatic verification of finite-state concurrent systems using temporal logic specifications. ACM Trans. Program. Lang. Syst., 8(2):244263, April 1986.
  9. Bonnie Collier, Tom DeMarco, and Peter Fearey. A defined process for project post mortem review. IEEE Software, 13(4):65-72, 1996.
  10. Juliet Corbin and Anselm Strauss. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Inc, 3 edition, 2007.
  11. Domenico Cotroneo, Michael Grottke, Roberto Natella, Roberto Pietran- tuono, and Kishor S. Trivedi. Fault triggers in open-source software: An experience report. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pages 178-187, 2013.
  12. Sandeep Dalal and Rajender Singh Chhillar. Case studies of most common and severe types of software system failure. International Journal of Advanced Research in Computer Science and Software Engineering, 2(8), 2012.
  13. Sandeep Dalal and Rajender Singh Chhillar. Empirical study of root cause analysis of software failure. ACM SIGSOFT Software Engineering Notes, 38(4):1-7, 2013.
  14. Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learn- ing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 17, page 12851298, New York, NY, USA, 2017. Association for Computing Machinery.
  15. Jan Eloff and Madeleine Bihina Bella. Software failures: An overview. In Software Failure Investigation, pages 7-24. Springer, 2018.
  16. Elmer Garduno, Soila P. Kavulya, Jiaqi Tan, Rajeev Gandhi, and Priya Narasimhan. Theia: Visual signatures for problem diagnosis in large hadoop clusters. In Presented as part of the 26th Large Installation System Administration Conference (LISA 12), pages 33-42, San Diego, CA, 2012. USENIX.
  17. Michael Grottke, Allen P. Nikora, and Kishor S. Trivedi. An empirical investigation of fault types in space mission system software. In 2010 IEEE/IFIP International Conference on Dependable Systems Networks (DSN), pages 447-456, 2010.
  18. Eben M. Haber and John Bailey. Design guidelines for system administration tools developed through ethnographic field studies. In Proceedings of the ACM Symposium on Computer human Interaction for Management of Information Technology, page 1, 2007.
  19. Shilin He, Jieming Zhu, Pinjia He, , and Michael R. Lyu. Experience report: System log analysis for anomaly detection. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pages 207-218, 2016.
  20. Herbert Hecht. Fault-tolerant software for real-time applications. ACM Computing Surveys (CSUR), 8(4):391-407, 1976.
  21. Qiao Huang, Xin Xia, and David Lo. Supervised vs unsupervised models: A holistic look at effort-aware just-in-time defect prediction. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 159-170, 2017.
  22. Tong Jia, Lin Yang, Pengfei Chen, Ying Li, Fanjing Meng, and Jingmin Xu. Logsed: Anomaly diagnosis through mining time-weighted control flow graph in logs. In 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), pages 447-455, June 2017.
  23. Lucas Layman, Madeline Diep, Meiyappan Nagappan, Janice Singer, Robert Deline, and Gina Venolia. Debugging revisited: Toward un- derstanding the debugging needs of contemporary software developers. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, pages 383-392, October 2013.
  24. Paul Luo Li, Andrew J. Ko, and Andrew Begel. Cross-disciplinary perspectives on collaborations with software engineers. In Proceedings of the 10th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 17, page 28. IEEE Press, 2017.
  25. Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. Log clustering based problem identification for online service systems. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE 16, page 102111, New York, NY, USA, 2016. Association for Computing Machinery.
  26. Tania Lorido-Botran, Jose Miguel-Alonso, and Jose A. Lozano. A review of auto-scaling techniques for elastic applications in cloud environments. Journal of Grid Computing, 12(4):559592, 2014.
  27. Shaun Phillips, Guenther Ruhe, and Jonathan Sillito. Information needs for integration decisions in the release process of large-scale parallel development. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work, New York, NY, USA, 2012. Association for Computing Machinery.
  28. William H Pierce. Failure-tolerant computer design. Academic Press, 1965.
  29. Sandra Rapps and Elaine J. Weyuker. Selecting software test data using data flow information. IEEE Transactions on Software Engineering, SE-11(4):367-375, 1985.
  30. Chris Schneider, Adam Barker, and Simon Dobson. A survey of self-healing systems frameworks. Software: Practice and Experience, 45(10):1375-1398, 2015.
  31. Kazumasa Shimari, Takashi Ishio, Tetsuya Kanda, and Katsuro Inoue. Near-omniscient debugging for java using size-limited execution trace. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 398-401, 2019.
  32. Jonathan Sillito, Gail C. Murphy, and Kris De Volder. Asking and answering questions during a programming change task. IEEE Trans- actions on Software Engineering, 34(4):434-451, July 2008.
  33. Nicole F. Velasquez and Suzanne P. Weisband. Work practices of system administrators: Implications for tool design. In Proceedings of the ACM Symposium on Computer human Interaction for Management of Information Technology, 2008.

Cited by

Sorted by citations

Authors
Sillito, J., & Kutomi, E.
Affiliations
Brigham Young University (US)
ORCID
Sillito, Jonathan
Venue
2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)
Publisher
IEEE
Published
2020-09
Pages
185-195
ISSN
1063-6773 (ISSN-L)
DOI
10.1109/icsme46990.2020.00027
OpenAlex
W3080443938
Cited by
14
References
2
Citations by year
2026: 1 · 2025: 2 · 2024: 5 · 2022: 2 · 2021: 4
FWCI
1.26
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Software Maintenance (ICSM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
Keywords
incident response, fault detection, failure mitigation, cascading failures, system outages, software maintenance, production support, software engineering practices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review