The nature of data center traffic: measurements & analysis
The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
We explore the nature of traffic in data centers, designed to support the mining of massive data sets. We instrument the servers to collect socket-level logs, with negligible performance impact. In a 1500 server operational cluster, we thus amass roughly a petabyte of measurements over two months, from which we obtain and report detailed views of traffic and congestion conditions and patterns. We further consider whether traffic matrices in the cluster might be obtained instead via tomographic inference from coarser-grained counter data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
MapReduce: Simplified Data Processing on Large Cluster
Senior Software Developer, A. U., & Dayalan, M. (2018) · International Journal of Research and Engineering · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,948
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
VL2: a scalable and flexible data center network
Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., & Sengupta, S. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592576 · Cited by 2,111
Self-citationBigtable: a distributed storage system for structured data
Chang, F. W., Dean, J. B., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2006) · doi:10.5555/1298455.1298475 · Cited by 1,928
Pig latin: a not-so-foreign language for data processing
Olston, C., Reed, B., Srivastava, U., Kumar, R., & Tomkins, A. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376726 · Cited by 1,730
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
Barroso, L. A., & Hölzle, U. (2009) · Synthesis lectures on computer architecture · doi:10.2200/s00193ed1v01y200905cac006 · Cited by 1,714
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · Proceedings of the nineteenth ACM symposium on Operating systems principles · doi:10.1145/945445.945450 · Cited by 1,423
SCOPE: easy and efficient parallel processing of massive data sets
Chaiken, R., Jenkins, B., Larson, P.-Å., Ramsey, B., Shakib, D., Weaver, S., & Zhou, J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454166 · Cited by 727
Self-citationGigascope: a stream database for network applications
Cranor, C., Johnson, T., Spataschek, O., & Shkapenyuk, V. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872838 · Cited by 687
Cited by
Network traffic characteristics of data centers in the wild
Benson, T., Akella, A., & Maltz, D. A. (2010) · Proceedings of the 10th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1879141.1879175 · Cited by 2,214
Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., & Sridharan, M. (2010) · Proceedings of the ACM SIGCOMM 2010 conference · doi:10.1145/1851182.1851192 · Cited by 1,877
Self-citationHedera: dynamic flow scheduling for data center networks
Al-Fares, M., Radhakrishnan, S., Raghavan, B., Huang, N., & Vahdat, A. M. (2010) · doi:10.5555/1855711.1855730 · Cited by 1,450
DevoFlow: scaling flow management for high-performance networks
Curtis, A. R., Mogul, J. C., Tourrilhes, J., Yalagandula, P., Sharma, P., & Banerjee, S. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018466 · Cited by 959
Inside the Social Network's (Datacenter) Network
Roy, A., Zeng, H., Bagga, J., Porter, G., & Snoeren, A. C. (2015) · Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication · doi:10.1145/2785956.2787472 · Cited by 849
Kambatla, K., Kollias, G., Kumar, V., & Grama, A. (2014) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2014.01.003 · Cited by 819
A survey on software-defined network and openflow: From concept to implementation
Hu, F., Hao, Q., & Bao, K. (2014) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2326417 · Cited by 752
Towards predictable datacenter networks
Ballani, H., Costa, P., Karagiannis, T., & Rowstron, A. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018465 · Cited by 750
Reining in the outliers in map-reduce clusters using Mantri
Ananthanarayanan, G., Kandula, S., Greenberg, A. G., Stoica, I. G., Lu, Y., Saha, B., & Harris, E. F. (2010) · Operating Systems Design and Implementation · doi:10.5555/1924943.1924962 · Cited by 636
Self-citationMicroTE: fine grained traffic engineering for data centers
Benson, T., Anand, A., Akella, A., & Zhang, M. (2011) · Proceedings of the Seventh COnference on emerging Networking EXperiments and Technologies · doi:10.1145/2079296.2079304 · Cited by 604
A Survey on Optical Interconnects for Data Centers
Kachris, C., & Tomkos, I. (2012) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2011.122111.00069 · Cited by 594
Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., & Sridharan, M. (2010) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1851275.1851192 · Cited by 548
Self-citationDevoFlow: scaling flow management for high-performance networks
Curtis, A. R., Mogul, J. C., Tourrilhes, J., Yalagandula, P., Sharma, P., & Banerjee, S. (2011) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2043164.2018466 · Cited by 543
Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection
Curtis, A. R., Kim, W., & Yalagandula, P. (2011) · 2011 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2011.5934956 · Cited by 402
Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
Guo, C., Yuan, L., Xiang, D., Dang, Y., Huang, R., Maltz, D., Liu, Z., Wang, V., Pang, B., Chen, H., et al. (2015) · Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication · doi:10.1145/2785956.2787496 · Cited by 390
A survey: Control plane scalability issues and approaches in Software-Defined Networking (SDN)
Karakus, M., & Durresi, A. (2017) · Computer Networks · doi:10.1016/j.comnet.2016.11.017 · Cited by 373
A Survey and a Layered Taxonomy of Software-Defined Networking
Jarraya, Y., Madi, T., & Debbabi, M. (2014) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2320094 · Cited by 372
Consolidating virtual machines with dynamic bandwidth demand in data centers
Wang, M., Meng, X., & Zhang, L. (2011) · 2011 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2011.5935254 · Cited by 339
Sharing the data center network
Shieh, A., Kandula, S., Greenberg, A. G., Kim, C., & Saha, B. (2011) · doi:10.5555/1972457.1972489 · Cited by 322
Self-citationAugmenting data center networks with multi-gigabit wireless links
Halperin, D., Kandula, S., Padhye, J., Bahl, P., & Wetherall, D. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018442 · Cited by 312
Self-citationAuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization
Chen, L., Lingys, J., Chen, K., & Liu, F. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230551 · Cited by 282
DRILL: Micro Load Balancing for Low-latency Data Center Networks
Ghorbani, S., Yang, Z., Godfrey, P. B., Ganjali, Y., & Firoozshahian, A. (2017) · Proceedings of the Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3098822.3098839 · Cited by 271
Mirror mirror on the ceiling: flexible wireless links for data centers
Zhou, X., Zhang, Z., Zhu, Y., Li, Y., Kumar, S., Vahdat, A., Zhao, B. Y., & Zheng, H. (2012) · Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication · doi:10.1145/2342356.2342440 · Cited by 270
A Survey on Data Center Networking (DCN): Infrastructure and Operations
Xia, W., Zhao, P., Wen, Y., & Xie, H. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2626784 · Cited by 267
Spatio-Temporal Compressive Sensing and Internet Traffic Matrices (Extended Version)
Roughan, M., Zhang, Y., Willinger, W., & Qiu, L. (2012) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2011.2169424 · Cited by 267
A taxonomy and survey on Green Data Center Networks
Bilal, K., Malik, S. U. R., Khalid, O., Hameed, A., Alvarez, E., Wijaysekara, V., Irfan, R., Shrestha, S., Dwivedy, D., Ali, M., et al. (2014) · Future Generation Computer Systems · doi:10.1016/j.future.2013.07.006 · Cited by 201
Datacenter Traffic Control: Understanding Techniques and Tradeoffs
Noormohammadpour, M., & Raghavendra, C. S. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2782753 · Cited by 172
“erate lots of internal datacenter traffic, such as scatter-gather (also known as partition-aggregate) [27]–[30] and batch computing tasks [31], [32].”
InfluentialOptimizing Virtual Machine placement in IaaS data centers: Taxonomy, review and open issues
Talebian, H., Gani, A., Sookhak, M., Abdelatif, A. A., Yousafzai, A., Vasilakos, A. V., & Yu, F. R. (2020) · Cluster Computing · doi:10.1007/s10586-019-02954-w · Cited by 72
Toward Delay-Efficient Game-Aware Data Centers for Cloud Gaming
Amiri, M., Osman, H. A., Shirmohammadi, S., & Abdallah, M. (2016) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/2983639 · Cited by 40
“An empirical study of a data center network consisting of 150 interswitches and 1500 servers has shown that 15% congestion situations can last for more than 100s, while many network links are often underutilized [Kandula et al. 2009].”
Network-Aware Virtual Machine Placement and Migration in Cloud Data Centers
Ferdaus, M. H., Murshed, M., Calheiros, R. N., & Buyya, R. (2015) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-4666-8213-9.ch002 · Cited by 38
SDN-based game-aware network management for cloud gaming
Amiri, M., Al Osman, H., Shirmohammadi, S., & Abdallah, M. (2015) · 2015 International Workshop on Network and Systems Support for Games (NetGames) · doi:10.1109/netgames.2015.7382989 · Cited by 10
“An empirical study on user tolerance for delay demonstrates that a delay of less than 100 msec is highly desirable for high action paced games, whereas 150 msec is the delay threshold in slow paced games [16].”
Background
Metadata
- Authors
- Kandula, S., Sengupta, S., Greenberg, A., Patel, P., & Chaiken, R.
- Venue
- Proceedings of the 9th ACM SIGCOMM conference on Internet measurement
- Publisher
- ACM
- Published
- 2009-11-04
- Pages
- 202-208
- OpenAlex
- W2062832101
- Cited by
- 1,030
- References
- 24
- FWCI
- 27.26
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.