Computation Scheduling for Distributed Machine Learning With Straggling Workers
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
We study scheduling of computation tasks across n workers in a large scale distributed learning problem with the help of a master. Computation and communication delays are assumed to be random, and redundant computations are assigned to workers in order to tolerate stragglers. We consider sequential computation of tasks assigned to a worker, while the result of each computation is sent to the master right after its completion. Each computation round, which can model an iteration of the stochastic gradient descent (SGD) algorithm, is completed once the master receives k distinct computations, referred to as the computation target. Our goal is to characterize the average completion time as a function of the computation load, which denotes the portion of the dataset available at each worker, and the computation target. We propose two computation scheduling schemes that specify the tasks assigned to each worker, as well as their computation schedule, i.e., the order of execution. Assuming a general statistical model for computation and communication delays, we derive the average completion time of the proposed schemes. We also establish a lower bound on the minimum average completion time by assuming prior knowledge of the random delays. Experimental results carried out on Amazon EC2 cluster show a significant reduction in the average completion time over existing coded and uncoded computing schemes. It is also shown numerically that the gap between the proposed scheme and the lower bound is relatively small, confirming the efficiency of the proposed scheduling design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134
Speeding Up Distributed Machine Learning Using Codes
Lee, K., Lam, M., Pedarsani, R., Papailiopoulos, D., & Ramchandran, K. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2736066 · Cited by 829
Federated Learning Over Wireless Fading Channels
Amiri, M. M., & Gunduz, D. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.2974748 · Cited by 625
Self-citationSparse Communication for Distributed Gradient Descent
Aji, A. F., & Heafield, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1045 · Cited by 585
Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852172 · Cited by 240
Research works printed but not linked: 36
- chen, 2017, AdaComp: Adaptive residual gradient compression for data-parallel distributed training, arXiv 1712 02679v1
- 10.1109/ALLERTON.2014.7028591 · DOI
- 10.1109/TII.2018.2851241 · DOI · Paper page
- choudhari, 0, Prioritized task scheduling in fog computing, Proc 34 Annual ACM Southeast Conf, 401
- 10.22266/ijies2018.0630.26 · doi:10.22266/ijies2018.0630.26
- 10.1016/S0022-0000(75)80008-0 · DOI · Paper page
- 10.1109/TNET.2019.2919553 · DOI
- 10.1109/ALLERTON.2017.8262882 · DOI
- tao, 0, eSGD: Communication efficient distributed deep learning on the edge, Proc Workshop Hot Topics Edge Comput
- yu, 2018, Polynomial codes: An optimal design for high-dimensional coded matrix multiplication, arXiv 1705 10464
- yu, 2018, Lagrange coded computing: Optimal design for resiliency, security and privacy, arXiv 1806 00939
- li, 2018, Polynomially coded regression: Optimal straggler mitigation via data encoding, arXiv 1805 09934v1
- 10.1109/ISIT.2018.8437473 · DOI
- 10.1109/ICASSP.2019.8682911 · DOI · Paper page
- 10.1109/IPDPSW.2018.00137 · DOI · Paper page
- 10.1109/ISIT.2018.8437871 · DOI
- mallick, 2018, Rateless codes for near-perfect load balancing in distributed matrix-vector multiplication, arXiv 1804 10332
- 10.1109/ISIT.2019.8849684 · DOI
- 10.4304/jsw.9.2.466-473 · doi:10.4304/jsw.9.2.466-473
- tandon, 0, Gradient coding: Avoiding stragglers in distributed learning, Proc Int Conf Mach Learn, 3368
- 10.1145/1478873.1478901 · DOI
- dutta, 2018, Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD, arXiv 1803 01113v3
- zhan, 2012, Improved PSO-based task scheduling algorithm in cloud computing, J Inf Comput Sci, 9, 3821
- halbawi, 2017, Improving distributed gradient descent using Reed-Solomon codes, arXiv 1706 05436
- ye, 2018, Communication-computation efficient gradient coding, arXiv 1802 03475v1
- 10.1109/ISIT.2019.8849514 · DOI
- 10.1002/cpe.4330060702 · doi:10.1002/cpe.4330060702
- 10.1109/ISIT.2018.8437563 · DOI
- 10.1561/2200000016 · DOI · Paper page
- 10.1109/ISIT.2019.8849334 · DOI · Paper page
- amiri, 0, Collaborative machine learning at the wireless edge with blind transmitters, Proc IEEE Global Conf Signal Inf Process
- seide1, 0, 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs, Proc INTERSPEECH, 1058
- wang, 0, ATOMO: Communication-efficient learning via atomic sparsification, Proc Int Conf Neural Inf Process, 9872
- konecny, 2017, Federated learning: Strategies for improving communication efficiency, arXiv 1610 05492
- attia, 2017, Combating computational heterogeneity in large-scale distributed computing via work exchange, arXiv 1711 08452
- strom, 0, Scalable distributed DNN training using commodity GPU cloud computing, Proc INTERSPEECH, 1488
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25
Federated Deep Learning for Immersive Virtual Reality over Wireless Networks
Chen, M., Semiari, O., Saad, W., Liu, X., & Yin, C. (2019) · 2019 IEEE Global Communications Conference (GLOBECOM) · doi:10.1109/globecom38437.2019.9013419 · Cited by 19
Metadata
- Authors
- Amiri, M. M., & Gunduz, D.
- Venue
- IEEE Transactions on Signal Processing
- Published
- 2019-12-15
- Volume
- 67
- Issue
- 24
- Pages
- 6270-6284
- ISSN
- 1053-587X (ISSN-L)
- 1941-0476
- OpenAlex
- W4249253882
- Cited by
- 65
- References
- 5
- FWCI
- 13.43
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- H2020 European Research Council (677854)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Computation Scheduling for Distributed Machine Learning With Straggling Workers. https://app.cochat.ai/papers/10.1109/tsp.2019.2952051
- The paper
- Amiri, M. M., & Gunduz, D. (2019). Computation Scheduling for Distributed Machine Learning With Straggling Workers. IEEE Transactions on Signal Processing. https://doi.org/10.1109/tsp.2019.2952051
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.