Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
Open accessThe DOI matches its OpenAlex record; 9 of the 9 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · cited 8×
“An algorithm of specific interest to solve problem (1) is mini-batch SGD (Dekel et al., 2012; Tak´aˇc et al., 2013), a parallel version of stochastic gradient descent (SGD) (Robbins & Monro, 1951; Nemirovski et al., 2009).”
BackgroundInfluentialAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 2×
“There are other approaches aiming to reduce the communication cost, such as increased mini-batch sizes (Goyal et al., 2017), defining local problems for each worker (Shamir et al., 2014; Jaggi et al., 2014; Ma et al., 2015; Reddi et al., 2016; Ma et al., 2017), reducing the communication frequency…”
BackgroundDeep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · cited 2×
“…as a heuristic to reduce communication cost, such as for instance in 1BitSGD (Seide et al., 2014; Strom, 2015) that rounds the gradient components to either -1 or 1, and error feedback schemes aim at reducing quantization errors among iterations, for instance as in (Lin et al., 2018).”
BackgroundStich, S. U., Cordonnier, J.-B., & Jaggi, M. (2018) · arXiv · doi:10.48550/arxiv.1809.07599 · Cited by 508
Self-citationCommunication-Efficient Distributed Optimization using an Approximate Newton-type Method
Shamir, O., Srebro, N., & Zhang, T. (2013) · arXiv · doi:10.48550/arxiv.1312.7853 · Cited by 424 · cited 2×
“…increasing the computation to communication ratio, such as increased mini-batch sizes (Goyal et al., 2017), defining local problems for each worker (Shamir et al., 2014; Reddi et al., 2016) or reducing the communication frequency (McDonald et al., 2009; Zinkevich et al., 2010; You et al., 2017;…”
BackgroundCommunication Compression for Decentralized Training
Tang, H., Gan, S., Zhang, C., Zhang, T., & Liu, J. (2018) · arXiv · doi:10.48550/arxiv.1803.06443 · Cited by 206
Randomized distributed mean estimation: Accuracy vs. communication
Konečný, J., & Richtárik, P. (2018) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2018.00062 · Cited by 69 · cited 2×
“Much sparser vectors can be obtained by random sparsification techniques that randomly mask the input vectors and only preserve a constant number of coordinates (Suresh et al., 2017; Konečný & Richtárik, 2016; Wangni et al., 2018; Stich et al., 2018).”
BackgroundSelf-citationFast distributed coordinate descent for non-strongly convex losses
Fercoq, O., Qu, Z., Richtarik, P., & Takac, M. (2014) · 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) · doi:10.1109/mlsp.2014.6958862 · Cited by 27
Cited by
UVeQFed: Universal Vector Quantization for Federated Learning
Shlezinger, N., Chen, M., Eldar, Y. C., Poor, H. V., & Cui, S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3046971 · Cited by 275
Communication-efficient distributed learning: An overview
Cao, X., Başar, T., Diggavi, S., Eldar, Y. C., Letaief, K. B., Poor, H. V., & Zhang, J. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3242710 · Cited by 139
vqSGD: Vector Quantized Stochastic Gradient Descent
Gandikota, V., Kane, D., Maity, R. K., & Mazumdar, A. (2019) · arXiv · doi:10.48550/arxiv.1911.07971 · Cited by 80
“In this work, we focus primarily on gradient quantization techniques, and note that the variance reduction techniques of [14] can be used on top of the proposed quantization schemes.”
BackgroundMethodCommunication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25
Metadata
- Authors
- Horváth, S., Kovalev, D. A., Mishchenko, K., Stich, S. U., & Richtárik, P.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2019-04-10
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.05115
- OpenAlex
- W2935990182
- Cited by
- 75
- References
- 9
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Stochastic Distributed Learning with Gradient Quantization and Variance Reduction. https://app.cochat.ai/papers/10.48550/arxiv.1904.05115
- The paper
- Horváth, S., Kovalev, D. A., Mishchenko, K., Stich, S. U., & Richtárik, P. (2019). Stochastic Distributed Learning with Gradient Quantization and Variance Reduction. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1904.05115
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.