The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Efficient coflow scheduling with Varys
Open accessAbstract
Communication in data-parallel applications often involves a collection of parallel flows. Traditional techniques to optimize flow-level metrics do not perform well in optimizing such collections, because the network is largely agnostic to application-level requirements. The recently proposed coflow abstraction bridges this gap and creates new opportunities for network scheduling. In this paper, we address inter-coflow scheduling for two different objectives: decreasing communication time of data-intensive jobs and guaranteeing predictable communication time. We introduce the concurrent open shop scheduling with coupled resources problem, analyze its complexity, and propose effective heuristics to optimize either objective. We present Varys, a system that enables data-intensive frameworks to use coflows and the proposed algorithms while maintaining high network utilization and guaranteeing starvation freedom. EC2 deployments and trace-driven simulations show that communication stages complete up to 3.16X faster on average and up to 2X more coflows meet their deadlines using Varys in comparison to per-flow mechanisms. Moreover, Varys outperforms non-preemptive coflow schedulers by more than 5X.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Self-citationA Bridging Model for Parallel Computation
Valiant, L. G. (1990) · ACM eBooks · doi:10.1145/3749740.3749753 · Cited by 3,664
Pregel: a system for large-scale graph processing
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., & Czajkowski, G. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807184 · Cited by 3,515
MapReduce: Simplified Data Processing on Large Cluster
Senior Software Developer, A. U., & Dayalan, M. (2018) · International Journal of Research and Engineering · doi:10.21276/ijre.2018.5.5.4 · Cited by 2,948
Dryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
VL2: a scalable and flexible data center network
Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., & Sengupta, S. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592576 · Cited by 2,111
(2011) · doi:10.5555/1972457.1972488 · Cited by 1,619
Self-citationHedera: dynamic flow scheduling for data center networks
Al-Fares, M., Radhakrishnan, S., Raghavan, B., Huang, N., & Vahdat, A. M. (2010) · doi:10.5555/1855711.1855730 · Cited by 1,450
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling
Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., & Stoica, I. (2010) · Proceedings of the 5th European conference on Computer systems · doi:10.1145/1755913.1755940 · Cited by 1,393
Self-citationPortLand: a scalable fault-tolerant layer 2 data center network fabric
Niranjan Mysore, R., Pamboris, A., Farrington, N., Huang, N., Miri, P., Radhakrishnan, S., Subramanya, V., & Vahdat, A. (2009) · Proceedings of the ACM SIGCOMM 2009 conference on Data communication · doi:10.1145/1592568.1592575 · Cited by 929
Critical-Path Planning and Scheduling: Mathematical Basis
Kelley, J. E. (1961) · Operations Research · doi:10.1287/opre.9.3.296 · Cited by 910
Achieving 100% throughput in an input-queued switch
McKeown, N., Mekkittikul, A., Anantharam, V., & Walrand, J. (1999) · IEEE Transactions on Communications · doi:10.1109/26.780463 · Cited by 838
Towards predictable datacenter networks
Ballani, H., Costa, P., Karagiannis, T., & Rowstron, A. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018465 · Cited by 750
SCOPE: easy and efficient parallel processing of massive data sets
Chaiken, R., Jenkins, B., Larson, P.-Å., Ramsey, B., Shakib, D., Weaver, S., & Zhou, J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454166 · Cited by 727
Lottery scheduling: flexible proportional-share resource management
Waldspurger, C. A., & Weihl, W. E. (1994) · doi:10.5555/1267638.1267639 · Cited by 637
Reining in the outliers in map-reduce clusters using Mantri
Ananthanarayanan, G., Kandula, S., Greenberg, A. G., Stoica, I. G., Lu, Y., Saha, B., & Harris, E. F. (2010) · Operating Systems Design and Implementation · doi:10.5555/1924943.1924962 · Cited by 636
Self-citationMicroTE: fine grained traffic engineering for data centers
Benson, T., Anand, A., Akella, A., & Zhang, M. (2011) · Proceedings of the Seventh COnference on emerging Networking EXperiments and Technologies · doi:10.1145/2079296.2079304 · Cited by 604
Critical-path planning and scheduling
Kelley, J. E., & Walker, M. R. (1959) · Papers presented at the December 1-3, 1959, eastern joint IRE-AIEE-ACM computer conference on - IRE-AIEE-ACM '59 (Eastern) · doi:10.1145/1460299.1460318 · Cited by 569
Managing data transfers in computer clusters with orchestra
Chowdhury, M., Zaharia, M., Ma, J., Jordan, M. I., & Stoica, I. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018448 · Cited by 543
Self-citationFinishing flows quickly with preemptive scheduling
Hong, C.-Y., Caesar, M., & Godfrey, P. B. (2012) · Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication · doi:10.1145/2342356.2342389 · Cited by 451
PortLand: a scalable fault-tolerant layer 2 data center network fabric
Niranjan Mysore, R., Pamboris, A., Farrington, N., Huang, N., Miri, P., Radhakrishnan, S., Subramanya, V., & Vahdat, A. (2009) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1594977.1592575 · Cited by 379
FairCloud: sharing the network in cloud computing
Popa, L., Kumar, G., Chowdhury, M., Krishnamurthy, A., Ratnasamy, S., & Stoica, I. (2012) · Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication · doi:10.1145/2342356.2342396 · Cited by 360
Self-citationCoflow: a networking abstraction for cluster applications
Chowdhury, M., & Stoica, I. (2012) · Proceedings of the 11th ACM Workshop on Hot Topics in Networks · doi:10.1145/2390231.2390237 · Cited by 340
Self-citationPACMan: coordinated memory caching for parallel jobs
Ananthanarayanan, G., Ghodsi, A., Wang, A., Borthakur, D., Kandula, S., Shenker, S. J., & Stoica, I. G. (2012) · Networked Systems Design and Implementation · Cited by 334
Self-citationSharing the data center network
Shieh, A., Kandula, S., Greenberg, A. G., Kim, C., & Saha, B. (2011) · doi:10.5555/1972457.1972489 · Cited by 322
The only constant is change: incorporating time-varying network reservations in data centers
Xie, D., Ding, N., Hu, Y., & Kompella, R. R. (2012) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2342356.2342397 · Cited by 282
Optimizing the "one big switch" abstraction in software-defined networks
Kang, N., Liu, Z., Rexford, J., & Walker, D. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535373 · Cited by 282
Participatory networking: an API for application control of SDNs
Ferguson, A. D., Guha, A., Liang, C., Fonseca, R., & Krishnamurthi, S. (2013) · Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM · doi:10.1145/2486001.2486003 · Cited by 269
Decentralized task-aware scheduling for data center networks
Dogar, F. R., Karagiannis, T., Ballani, H., & Rowstron, A. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2626322 · Cited by 247
Re-optimizing data-parallel computing
Agarwal, S., Kandula, S., Bruno, N., Wu, M.-C., Stoica, I. G., & Zhou, J. (2012) · UC Berkeley · Cited by 206
Self-citationLeveraging endpoint flexibility in data-intensive clusters
Chowdhury, M., Kandula, S., & Stoica, I. G. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2486021 · Cited by 162
Self-citationAn Architecture for Internet Data Transfer
Tolia, N., Kaminsky, M., Andersen, D. G., & Patil, S. (2018) · doi:10.1184/r1/6603203 · Cited by 79
Cited by
A generic communication scheduler for distributed DNN training acceleration
Peng, Y., Zhu, Y., Chen, Y., Bao, Y., Yi, B., Lan, C., Wu, C., & Guo, C. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359642 · Cited by 321
AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization
Chen, L., Lingys, J., Chen, K., & Liu, F. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230551 · Cited by 282
Low Latency Geo-distributed Data Analytics
Pu, Q., Ananthanarayanan, G., Bodik, P., Kandula, S., Akella, A., Bahl, P., & Stoica, I. (2015) · Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication · doi:10.1145/2785956.2787505 · Cited by 246
Self-citation
Metadata
- Authors
- Chowdhury, M., Zhong, Y., & Stoica, I.
- Venue
- Proceedings of the 2014 ACM conference on SIGCOMM
- Publisher
- ACM
- Published
- 2014-08-17
- Pages
- 443-454
- OpenAlex
- W2149939304
- Cited by
- 438
- References
- 32
- FWCI
- 39.90
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.