Understanding network failures in data centers: measurement, analysis, and implications
Open accessAbstract
We present the first large-scale analysis of failures in a data center network. Through our analysis, we seek to answer several fundamental questions: which devices/links are most unreliable, what causes failures, how do failures impact network traffic and how effective is network redundancy? We answer these questions using multiple data sources commonly collected by network operators. The key findings of our study are that (1) data center networks show high reliability, (2) commodity switches such as ToRs and AggS are highly reliable, (3) load balancers dominate in terms of failure occurrences with many short-lived software related faults,(4) failures have potential to cause loss of many small packets such as keep alive messages and ACKs, and (5) network redundancy is only 40% effective in reducing the median impact of failure.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Logs remain the most common operational signal, and also one of the least forgiving.
“Logs remain the most common operational signal, and also one of the least forgiving [62], [63], [127], [128].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
The Introduction and §3.3 describe network logs as noisy and sometimes inconsistent, supporting the point that they are difficult to use. But the paper does not establish that logs are the most common operational signal or compare them with other signals to show they are among the least forgiving. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredIn real deployments, the earlier promise of deep sequence models depends on good log quality, stable parsing, manageable workload shift, and templates that do not change faster than the detector can learn them.
“In real deployments, however, this earlier promise depends on rather plain conditions: good log quality, stable parsing, manageable workload shift, and templates that do not change faster than the detector can learn them [127], [129], [64].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
The Introduction and §3.3 describe noisy network logs and the need to filter and clean them, supporting the general point about log quality. The paper studies data-center network failures; it does not address deep sequence detectors, stable parsing, workload shift, or changing templates. (LLM, conf=0.84)
Experimental — accuracy not yet measuredIf the agent is trained or evaluated on cleanly parsed logs, but deployed on noisy and drifting templates, autonomy will be miscalibrated.
“For agentic AIOps, the implication is direct: if the agent is trained or evaluated on cleanly parsed logs, but deployed on noisy and drifting templates, autonomy will be miscalibrated [127], [128].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
The Introduction describes noisy network logs, and §3.3 describes cleaning inconsistent events before analysis. The paper does not address log-template drift, agent training or evaluation, or whether deployment on different logs miscalibrates autonomy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn real systems, log anomaly detection encounters missing events, changing templates, and unclear labels.
“In real systems, log anomaly detection encounters missing events, changing templates, and unclear labels [127].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
The Limitations section says failure reporting can miss events and that human bias can affect failure annotations; the Introduction also describes trouble-ticket annotations as ambiguous. The paper does not report changing log templates, and its discussion concerns analysis of network failures rather than log anomaly detection generally. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe state observed by an agent can be skewed due to a partial view.
“The state observed by an agent can be skewed due to a partial view [57], [127], [137], [138].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
Drift is the default case in operations.
“Drift is the default case in operations [127], [137], [138], [”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [127]
Section 4.3 mentions configuration errors among causes of network failures, and the discussion of network redundancy describes one configuration typo. Neither passage says that drift is the default in operations. Searches for drift and for routine configuration changes found no passage establishing that claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 5
- D. Ford, F. Labelle, F. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan. Availability in globally distributed storage systems. In OSDI, 2010.
- C. Labovitz and A. Ahuja. Experimental study of internet stability and wide-area backbone failures. In The Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing, 1999.
- A. Markopoulou, G. Iannaccone, S. Bhattacharyya, C.-N. Chuah, Y. Ganjali, and C. Diot. Characterization of failures in an operational IP backbone network. IEEE/ACM Transactions on Networking, 2008.
- A. Shaikh, C. Isett, A. Greenberg, M. Roughan, and J. Gottlieb. A case study of OSPF behavior in a large enterprise network. In ACM IMW, 2002.
- K. V. Vishwanath and N. Nagappan. Characterizing cloud computing hardware reliability. In Symposium on Cloud Computing (SOCC), 2010.
Cited by
Sorted by citations
Alur, R., Bodik, R., Juniwal, G., Martin, M. M. K., Raghothaman, M., Seshia, S. A., Singh, R., Solar-Lezama, A., Torlak, E., & Udupa, A. (2013) · 2013 Formal Methods in Computer-Aided Design · doi:10.1109/fmcad.2013.6679385 · Cited by 656 · see citing works
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075
- Authors
- Gill, P., Jain, N., & Nagappan, N.
- Affiliations
- University of Toronto (CA)
- Microsoft (United States) (US)
- Venue
- Proceedings of the ACM SIGCOMM 2011 conference
- Publisher
- ACM
- Published
- 2011-08-15
- Pages
- 350-361
- OpenAlex
- W2126969025
- Cited by
- 714
- References
- 0
- Citations by year
- 2026: 11 · 2025: 26 · 2024: 21 · 2023: 40 · 2022: 25 · 2021: 56 · 2020: 67 · 2019: 57 · 2018: 63 · 2017: 75 · 2016: 85 · 2015: 64 · 2014: 40 · 2013: 59 · 2012: 23
- FWCI
- 35.04
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGCOMM Conference (SIGCOMM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
- Keywords
- data center networks, network failures, failure analysis, network reliability, network redundancy, load balancer, software defects, commodity switches
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review