Decentralized Stochastic Optimization and Machine Learning: A Unified Variance-Reduction Framework for Robust Performance and Fast Convergence
The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Abstract
Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data samples are distributed across a network of nodes, and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic first-order methods and provide a unified algorithmic framework that combines variance reduction with gradient tracking to achieve robust performance and fast convergence. We provide explicit theoretical guarantees of the corresponding methods when the objective functions are smooth and strongly convex and show their applicability to nonconvex problems via numerical experiments. Throughout the article, we provide intuitive illustrations of the main technical ideas by casting appropriate tradeoffs and comparisons among the methods of interest and by highlighting applications to decentralized training of machine learning models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Specifically, the variance reduction with the gradient tracking method was investigated in [146] to achieve a fast convergence rate.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [146]
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066 · cited 3×
“…where the objective functions are smooth and strongly convex [2]; text classification with support vector machines, where the objectives are convex but not necessarily smooth [1]; and perception tasks with deep neural networks, where the cost functions are nonconvex, in general [1], [2].”
BackgroundMethodInfluentialCommunication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 1×
“Various programming models and several variants of master–worker configurations that are tailored for specific computing needs and environments have been proposed, such as MapReduce, AllReduce, and federated learning [50].”
BackgroundConsensus and Cooperation in Networked Multi-Agent Systems
Olfati-Saber, R., Fax, J. A., & Murray, R. M. (2007) · Proceedings of the IEEE · doi:10.1109/jproc.2006.887293 · Cited by 10,522 · cited 3×
“…, ; i 6 and 2) optimality—the agreement should be on the minimizer of the global objective F , i.e., . cons i i = ) Average-consensus algorithms [10] are information fusion protocols that enable each node to appropriately combine the vectors received from its neighbors and agree on the average…”
BackgroundMethodResultInfluentialHorn, R. A., & Johnson, C. R. (2012) · doi:10.1017/cbo9781139020411 · Cited by 4,642
Distributed Subgradient Methods for Multi-Agent Optimization
Nedic, A., & Ozdaglar, A. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.2009515 · Cited by 3,719 · cited 1×
“A well-known solution to this problem is DGD [13], [14], described as follows.”
MethodOptimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Gossip-based computation of aggregate information
Kempe, D., Dobra, A., & Gehrke, J. (2004) · 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. · doi:10.1109/sfcs.2003.1238221 · Cited by 1,480 · cited 1×
“A well-studied solution to this issue is based on the push-sum (type) algorithms [39] that enable consensus with non-DS weights through the help of eigenvector estimation.”
MethodPattern Recognition and Machine Learning
Maindonald, J. (2007) · Journal of Statistical Software · doi:10.18637/jss.v017.b05 · Cited by 1,349
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
Shi, W., Ling, Q., Wu, G., & Yin, W. (2015) · SIAM Journal on Optimization · doi:10.1137/14096668x · Cited by 1,261
Research works printed but not linked: 1
- lecun, 1998, The MNIST Database of Handwritten Digits
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Specifically, the variance reduction with the gradient tracking method was investigated in [146] to achieve a fast convergence rate.”
MethodTwelve Scientific Challenges for 6G: Rethinking the Foundations of Communications Theory
Chafii, M., Bariah, L., Muhaidat, S., & Debbah, M. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3243918 · Cited by 289
“With the aim to enable self-optimizing large-scale networks, decentralized stochastic optimization is proposed as an efficient method to realize network-wide optimization, with leveraging a large number of resources-constrained devices which communicate with each other in a decentralized ad-hoc manner [126].”
MethodFederated Learning Over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis
Xing, H., Simeone, O., & Bi, S. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118400 · Cited by 160
“There are important scenarios in which a base station-centric architecture is either unavailable or undesirable due to coverage, privacy, implementation efficiency, or fault-tolerance considerations [14], [15].”
BackgroundA General Framework for Decentralized Optimization With First-Order Methods
Xin, R., Pu, S., Nedić, A., & Khan, U. A. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3024266 · Cited by 149
“Remark 19 (Other instances of the GT-VR framework): The discussion in this section enables a suite of methods that combine various gradient tracking and variance reduction techniques leading to different performance and implementation trade-offs towards different problem classes [21], [69].”
BackgroundMethodInfluentialSelf-citationAn Improved Convergence Analysis for Decentralized Online Stochastic Non-Convex Optimization
Xin, R., Khan, U. A., & Kar, S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3062553 · Cited by 147
“P For further intuition and explanation, see [2], [10], for example.”
BackgroundSelf-citation
Metadata
- Authors
- Xin, R., Kar, S., & Khan, U. A.
- Affiliations
- Carnegie Mellon University (US)
- Tufts University (US)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2020-05
- Volume
- 37
- Issue
- 3
- Pages
- 102-113
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3023944502
- Cited by
- 114
- References
- 53
- Citations by year
- 2026: 5 · 2025: 15 · 2024: 13 · 2023: 16 · 2022: 28 · 2021: 22 · 2020: 13
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Funders
- National Science Foundation (CCF-1350264; CCF-1513936; CMMI-1903972; CBET-1935555)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- distributed optimization, stochastic optimization, variance reduction, gradient tracking, finite-sum minimization, decentralized machine learning, decentralized learning, non-convex optimization, strong convexity, convergence guarantees, privacy-preserving machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Decentralized Stochastic Optimization and Machine Learning: A Unified Variance-Reduction Framework for Robust Performance and Fast Convergence. https://app.cochat.ai/papers/10.1109/msp.2020.2974267
- The paper
- Xin, R., Kar, S., & Khan, U. A. (2020). Decentralized Stochastic Optimization and Machine Learning: A Unified Variance-Reduction Framework for Robust Performance and Fast Convergence. IEEE Signal Processing Magazine. https://doi.org/10.1109/msp.2020.2974267
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.