The DOI matches its Crossref record.
Automating Chaos Experiments in Production
Open accessAbstract
Distributed systems often face transient errors and localized component degradation and failure. Verifying that the overall system remains healthy in the face of such failures is challenging. At Netflix, we have built a platform for automatically generating and executing chaos experiments, which check how well the production system can handle component failures and slowdowns. This paper describes the platform and our experiences operating it.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, causal models have been used to estimate the impact of handover-parameter adjustments and other cellular-network configuration interventions, including transmission-power and cell-offset changes, on subsequent service and performance time series [121], [122], [123].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [123]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 22
- madappa, 2013, Announcing EVCache Distributed in-memory datastore for cloud
- probst, 2016, Engineering trade-offs and the netflix API re-architecture
- burns, 2018, Continuous Delivery With Spinnaker
- humble, 2010, Continuous Delivery Reliable Software Releases Through Build Test and Deployment Automation
- ranganathan, 2012, Netflix shares cloud load balancing and failover tool Eureka!
- harrington, 2014, Introducing Atlas: Netflix's primary telemetry platform, The Netflix Tech Blog
- schmaus, 2016, Stream-processing with Mantis, The Netflix Tech Blog
- cohen, 2013, Announcing Zuul: Edge service in the cloud, The Netflix Tech Blog
- graff, 2018, Automated canary analysis at netflix with kayenta, The Netflix Tech Blog
- rosenthal, 2017, Chaos Engineering Building Confidence in System Behavior through Experiments
- 10.1109/MS.2016.60 · DOI
- reynolds, 2016, Vizceral open source, The Netflix Tech Blog
- newman, 2015, Building Microservices
- 10.1109/ISSREW.2016.52 · DOI
- christensen, 2012, Introducing Hystrix for resilience engineering, The Netflix Tech Blog
- nygard, 2018, Release It! Design and Deploy Production-Ready Software
- gunawi, 0, What bugs live in the cloud? a study of 3000+ issues in cloud systems, Proc ACM Symposium on Cloud Computing (SOCC)
- andrus, 2014, FIT: Failure injection testing, The Netflix Tech Blog
- meshenberg, 2015, Active-active for multiregional resiliency, The Netflix Tech Blog
- posta, 2018, Introducing Istio Service Mesh for Microservices
- sigelman, 2010, Dapper, a large-scale distributed systems tracing infrastructure, Google Inc Tech Rep
- 10.1109/ICWS.2017.75 · DOI
Cited by
Faster, deeper, easier: crowdsourcing diagnosis of microservice kernel failure from user space
Pan, Y., Ma, M., Jiang, X., & Wang, P. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464805 · Cited by 42
DyCause: Crowdsourcing to Diagnose Microservice Kernel Failure
Pan, Y., Ma, M., Jiang, X., & Wang, P. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3233915 · Cited by 11
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
Metadata
- Authors
- Basiri, A., Hochstein, L., Jones, N., & Tucker, H.
- Affiliations
- Netflix (United States) (US)
- ORCID
- Hochstein, Lorin
- Venue
- 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 31-40
- OpenAlex
- W2944265102
- Cited by
- 37
- References
- 0
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 3 · 2023: 6 · 2022: 2 · 2021: 3 · 2020: 4
- FWCI
- 1.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- chaos engineering, distributed systems, fault tolerance, component failure, transient faults, production system reliability, Netflix, fault injection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
