CoChat

Computation Scheduling for Distributed Machine Learning With Straggling Workers

Open access
Amiri, Mohammad Mohammadi, Gunduz, Deniz · 2019 · IEEE Transactions on Signal Processing
Cited by 65

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

We study scheduling of computation tasks across n workers in a large scale distributed learning problem with the help of a master. Computation and communication delays are assumed to be random, and redundant computations are assigned to workers in order to tolerate stragglers. We consider sequential computation of tasks assigned to a worker, while the result of each computation is sent to the master right after its completion. Each computation round, which can model an iteration of the stochastic gradient descent (SGD) algorithm, is completed once the master receives k distinct computations, referred to as the computation target. Our goal is to characterize the average completion time as a function of the computation load, which denotes the portion of the dataset available at each worker, and the computation target. We propose two computation scheduling schemes that specify the tasks assigned to each worker, as well as their computation schedule, i.e., the order of execution. Assuming a general statistical model for computation and communication delays, we derive the average completion time of the proposed schemes. We also establish a lower bound on the minimum average completion time by assuming prior knowledge of the random delays. Experimental results carried out on Amazon EC2 cluster show a significant reduction in the average completion time over existing coded and uncoded computing schemes. It is also shown numerically that the gap between the proposed scheme and the lower bound is relatively small, confirming the efficiency of the proposed scheduling design.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tsp.2019.2952051

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 36

  1. chen, 2017, AdaComp: Adaptive residual gradient compression for data-parallel distributed training, arXiv 1712 02679v1
  2. 10.1109/ALLERTON.2014.7028591 · DOI
  3. 10.1109/TII.2018.2851241 · DOI · Paper page
  4. choudhari, 0, Prioritized task scheduling in fog computing, Proc 34 Annual ACM Southeast Conf, 401
  5. 10.22266/ijies2018.0630.26 · doi:10.22266/ijies2018.0630.26
  6. 10.1016/S0022-0000(75)80008-0 · DOI · Paper page
  7. 10.1109/TNET.2019.2919553 · DOI
  8. 10.1109/ALLERTON.2017.8262882 · DOI
  9. tao, 0, eSGD: Communication efficient distributed deep learning on the edge, Proc Workshop Hot Topics Edge Comput
  10. yu, 2018, Polynomial codes: An optimal design for high-dimensional coded matrix multiplication, arXiv 1705 10464
  11. yu, 2018, Lagrange coded computing: Optimal design for resiliency, security and privacy, arXiv 1806 00939
  12. li, 2018, Polynomially coded regression: Optimal straggler mitigation via data encoding, arXiv 1805 09934v1
  13. 10.1109/ISIT.2018.8437473 · DOI
  14. 10.1109/ICASSP.2019.8682911 · DOI · Paper page
  15. 10.1109/IPDPSW.2018.00137 · DOI · Paper page
  16. 10.1109/ISIT.2018.8437871 · DOI
  17. mallick, 2018, Rateless codes for near-perfect load balancing in distributed matrix-vector multiplication, arXiv 1804 10332
  18. 10.1109/ISIT.2019.8849684 · DOI
  19. 10.4304/jsw.9.2.466-473 · doi:10.4304/jsw.9.2.466-473
  20. tandon, 0, Gradient coding: Avoiding stragglers in distributed learning, Proc Int Conf Mach Learn, 3368
  21. 10.1145/1478873.1478901 · DOI
  22. dutta, 2018, Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD, arXiv 1803 01113v3
  23. zhan, 2012, Improved PSO-based task scheduling algorithm in cloud computing, J Inf Comput Sci, 9, 3821
  24. halbawi, 2017, Improving distributed gradient descent using Reed-Solomon codes, arXiv 1706 05436
  25. ye, 2018, Communication-computation efficient gradient coding, arXiv 1802 03475v1
  26. 10.1109/ISIT.2019.8849514 · DOI
  27. 10.1002/cpe.4330060702 · doi:10.1002/cpe.4330060702
  28. 10.1109/ISIT.2018.8437563 · DOI
  29. 10.1561/2200000016 · DOI · Paper page
  30. 10.1109/ISIT.2019.8849334 · DOI · Paper page
  31. amiri, 0, Collaborative machine learning at the wireless edge with blind transmitters, Proc IEEE Global Conf Signal Inf Process
  32. seide1, 0, 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs, Proc INTERSPEECH, 1058
  33. wang, 0, ATOMO: Communication-efficient learning via atomic sparsification, Proc Int Conf Neural Inf Process, 9872
  34. konecny, 2017, Federated learning: Strategies for improving communication efficiency, arXiv 1610 05492
  35. attia, 2017, Combating computational heterogeneity in large-scale distributed computing via work exchange, arXiv 1711 08452
  36. strom, 0, Scalable distributed DNN training using commodity GPU cloud computing, Proc INTERSPEECH, 1488

Cited by

See all 65

Metadata

Authors
Amiri, M. M., & Gunduz, D.
Venue
IEEE Transactions on Signal Processing
Published
2019-12-15
Volume
67
Issue
24
Pages
6270-6284
ISSN
  • 1053-587X (ISSN-L)
  • 1941-0476
OpenAlex
W4249253882
Cited by
65
References
5
FWCI
13.43
Type
journal article
Language
en
Open access
green
Funders
H2020 European Research Council (677854)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Computation Scheduling for Distributed Machine Learning With Straggling Workers. https://app.cochat.ai/papers/10.1109/tsp.2019.2952051
The paper
Amiri, M. M., & Gunduz, D. (2019). Computation Scheduling for Distributed Machine Learning With Straggling Workers. IEEE Transactions on Signal Processing. https://doi.org/10.1109/tsp.2019.2952051

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.