The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Decentralized Stochastic Optimization and Machine Learning: A Unified Variance-Reduction Framework for Robust Performance and Fast Convergence
Abstract
Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data samples are distributed across a network of nodes, and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic first-order methods and provide a unified algorithmic framework that combines variance reduction with gradient tracking to achieve robust performance and fast convergence. We provide explicit theoretical guarantees of the corresponding methods when the objective functions are smooth and strongly convex and show their applicability to nonconvex problems via numerical experiments. Throughout the article, we provide intuitive illustrations of the main technical ideas by casting appropriate tradeoffs and comparisons among the methods of interest and by highlighting applications to decentralized training of machine learning models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Specifically, the variance reduction with the gradient tracking method was investigated in [146] to achieve a fast convergence rate.”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [146]
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066 · cited 3×
“…where the objective functions are smooth and strongly convex [2]; text classification with support vector machines, where the objectives are convex but not necessarily smooth [1]; and perception tasks with deep neural networks, where the cost functions are nonconvex, in general [1], [2].”
BackgroundMethodInfluentialCommunication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 1×
“Various programming models and several variants of master–worker configurations that are tailored for specific computing needs and environments have been proposed, such as MapReduce, AllReduce, and federated learning [50].”
BackgroundConsensus and Cooperation in Networked Multi-Agent Systems
Olfati-Saber, R., Fax, J. A., & Murray, R. M. (2007) · Proceedings of the IEEE · doi:10.1109/jproc.2006.887293 · Cited by 10,522 · cited 3×
“…, ; i 6 and 2) optimality—the agreement should be on the minimizer of the global objective F , i.e., . cons i i = ) Average-consensus algorithms [10] are information fusion protocols that enable each node to appropriately combine the vectors received from its neighbors and agree on the average…”
BackgroundMethodResultInfluentialHorn, R. A., & Johnson, C. R. (2012) · doi:10.1017/cbo9781139020411 · Cited by 4,642
Distributed Subgradient Methods for Multi-Agent Optimization
Nedic, A., & Ozdaglar, A. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.2009515 · Cited by 3,719 · cited 1×
“A well-known solution to this problem is DGD [13], [14], described as follows.”
MethodOptimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Gossip-based computation of aggregate information
Kempe, D., Dobra, A., & Gehrke, J. (2004) · 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. · doi:10.1109/sfcs.2003.1238221 · Cited by 1,480 · cited 1×
“A well-studied solution to this issue is based on the push-sum (type) algorithms [39] that enable consensus with non-DS weights through the help of eigenvector estimation.”
MethodPattern Recognition and Machine Learning
Maindonald, J. (2007) · Journal of Statistical Software · doi:10.18637/jss.v017.b05 · Cited by 1,349
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
Shi, W., Ling, Q., Wu, G., & Yin, W. (2015) · SIAM Journal on Optimization · doi:10.1137/14096668x · Cited by 1,261
Research works printed but not linked: 1
- lecun, 1998, The MNIST Database of Handwritten Digits
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“Specifically, the variance reduction with the gradient tracking method was investigated in [146] to achieve a fast convergence rate.”
MethodTwelve Scientific Challenges for 6G: Rethinking the Foundations of Communications Theory
Chafii, M., Bariah, L., Muhaidat, S., & Debbah, M. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3243918 · Cited by 289
“With the aim to enable self-optimizing large-scale networks, decentralized stochastic optimization is proposed as an efficient method to realize network-wide optimization, with leveraging a large number of resources-constrained devices which communicate with each other in a decentralized ad-hoc manner [126].”
MethodFederated Learning Over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis
Xing, H., Simeone, O., & Bi, S. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118400 · Cited by 160
“There are important scenarios in which a base station-centric architecture is either unavailable or undesirable due to coverage, privacy, implementation efficiency, or fault-tolerance considerations [14], [15].”
BackgroundA General Framework for Decentralized Optimization With First-Order Methods
Xin, R., Pu, S., Nedić, A., & Khan, U. A. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3024266 · Cited by 148
“Remark 19 (Other instances of the GT-VR framework): The discussion in this section enables a suite of methods that combine various gradient tracking and variance reduction techniques leading to different performance and implementation trade-offs towards different problem classes [21], [69].”
BackgroundMethodInfluentialSelf-citationAn Improved Convergence Analysis for Decentralized Online Stochastic Non-Convex Optimization
Xin, R., Khan, U. A., & Kar, S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3062553 · Cited by 147
“P For further intuition and explanation, see [2], [10], for example.”
BackgroundSelf-citationIntegrating LLMs With ITS: Recent Advances, Potentials, Challenges, and Future Directions
Mahmud, D., Hajmohamed, H., Almentheri, S., Alqaydi, S., Aldhaheri, L., Khalil, R. A., & Saeed, N. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3528116 · Cited by 138
“The work by [95] provides a unified framework for decentralized stochastic optimization, emphasizing variance-reduction techniques to improve robustness and convergence speed, which are crucial for decentralized learning.”
Variance-Reduced Decentralized Stochastic Optimization With Accelerated Convergence
Xin, R., Khan, U. A., & Kar, S. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3031071 · Cited by 128
“Specifically, SAGA in practice demonstrates faster convergence compared with SVRG [7], [15], however at the expense of additional storage requirements.”
BackgroundResultSelf-citationSurvey of distributed algorithms for resource allocation over multi-agent systems
Doostmohammadian, M., Aghasi, A., Pirani, M., Nekouei, E., Zarrabi, H., Keypour, R., Rikos, A. I., & Johansson, K. H. (2024) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2024.100983 · Cited by 109
Zhou, G., Pan, C., Ren, H., Wang, K., Elkashlan, M., & Renzo, M. D. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3049257 · Cited by 98
Quantization Enabled Privacy Protection in Decentralized Stochastic Optimization
Wang, Y., & Başar, T. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3198030 · Cited by 79
“Furthermore, in the era of Internet of things which connect massive low-cost sensing and communication devices, the data fed to optimization computations are usually subject to measurement noises [8].”
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!Preprint
Kovalev, D. A., Koloskova, A., Jaggi, M., Richtárik, P., & Stich, S. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01697 · Cited by 75
“Wang and Joshi, 2018; Koloskova et al., 2020b), see also the review articles (Sayed, 2014; Xin et al., 2020; Nedić, 2020).”
BackgroundFast Decentralized Nonconvex Finite-Sum Optimization with Recursive Variance Reduction
Xin, R., Khan, U. A., & Kar, S. (2022) · SIAM Journal on Optimization · doi:10.1137/20m1361158 · Cited by 68
“This formulation often appears in decentralized empirical risk minimization, where each local cost fi can be considered as an empirical risk computed over a finite number of m local data samples [48], and lies at the heart of many modern machine learning problems [4, 22, 52].”
Self-citationPrimal–Dual Methods for Large-Scale and Distributed Convex Optimization and Data Analytics
Jakovetić, D., Bajović, D., Xavier, J., & Moura, J. M. F. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3007395 · Cited by 67
“Additional very recent works include [27], [28], [29].”
BackgroundOver-the-Air Decentralized Federated Learning
Shi, Y., Zhou, Y., & Shi, Y. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9517780 · Cited by 67
S-ADDOPT: Decentralized Stochastic First-Order Optimization Over Directed Graphs
Qureshi, M. I., Xin, R., Kar, S., & Khan, U. A. (2020) · IEEE Control Systems Letters · doi:10.1109/lcsys.2020.3006420 · Cited by 61
“However, all of these decentralized stochastic algorithms are built on undirected graphs, see [19] for a friendly tutorial.”
BackgroundSelf-citationDistributed Learning for Wireless Communications: Methods, Applications and Challenges
Qian, L., Yang, P., Xiao, M., Dobre, O. A., Renzo, M. D., Li, J., Han, Z., Qin, Y., & Zhao, J. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3156756 · Cited by 60
Decentralized Stochastic Optimization With Inherent Privacy Protection
Wang, Y., & Poor, H. V. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3174187 · Cited by 59
“Furthermore, with the advent of Internet of things, which features the explosive growth of low-cost sensing and communication devices, the data fed to distributed optimization are usually subject to measurement noises [5].”
BackgroundWireless for Machine Learning: A Survey
Hellström, H., da Silva, J. M. B., Amiri, M. M., Chen, M., Fodor, V., Poor, H. V., & Fischione, C. (2022) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000114 · Cited by 59
Byzantine-robust decentralized stochastic optimization over static and time-varying networks
Peng, J., Li, W., & Ling, Q. H. (2021) · Signal Processing · doi:10.1016/j.sigpro.2021.108020 · Cited by 59
Wireless for Machine Learning: A Survey
Hellström, H., da Silva, J. M. B., Amiri, M. M., Chen, M., Fodor, V., Poor, H. V., & Fischione, C. (2022) · doi:10.1561/9781638280071 · Cited by 57
Wang, Y., & Başar, T. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3212006 · Cited by 53
“Furthermore, in the era of Internet of Things, which connect massive low-cost sensing and communication devices, the data fed to optimization computations are usually subject to measurement noises [48].”
BackgroundAccelerated First-Order Optimization Algorithms for Machine Learning
Li, H., Fang, C., & Lin, Z. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3007634 · Cited by 52
“[96] implemented gradient tracking in SVRG (see [97] for a detailed review).”
A Fast Randomized Incremental Gradient Method for Decentralized Nonconvex Optimization
Xin, R., Khan, U. A., & Kar, S. (2021) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2021.3122586 · Cited by 50
Self-citationNetworked Signal and Information Processing: Learning by multiagent systems
Vlaski, S., Kar, S., Sayed, A. H., & Moura, J. M. F. (2023) · IEEE Signal Processing Magazine · doi:10.1109/msp.2023.3267896 · Cited by 47
“More recently, other variants of these procedures that employ techniques such as variance reduction and gradient tracking have been employed [54,55] (see also discussion below) that aim to combine the best of both worlds at the cost of more complex distributed update rules.”
MethodSelf-citationDistributed Support Vector Machines Over Dynamic Balanced Directed Networks
Doostmohammadian, M., Aghasi, A., Charalambous, T., & Khan, U. A. (2021) · IEEE Control Systems Letters · doi:10.1109/lcsys.2021.3086388 · Cited by 46
Self-citationDC-DistADMM: ADMM Algorithm for Constrained Optimization Over Directed Graphs
Khatana, V., & Salapaka, M. V. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3221856 · Cited by 44
“For optimization problems that are unconstrained [with no equality, inequality and set constraints in problem (1)], examples [8], [9], [10], [11], [12] (and references therein) take a primal based approach, whereas [13], [14], [15], [16], [17], [18], [19], [20] take a dual based approach.”
Decentralized bilevel optimization
Chen, X., Huang, M., & Ma, S. (2024) · Optimization Letters · doi:10.1007/s11590-024-02101-4 · Cited by 43
“We refer the interested readers to [37], which provides a comprehensive review of decentralized optimization in a unified variance reduction framework.”
BackgroundInfluentialFully Asynchronous Distributed Optimization with Linear Convergence in Directed NetworksPreprint
Zhang, J., & You, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08215 · Cited by 43
Distributed Anytime-Feasible Resource Allocation Subject to Heterogeneous Time-Varying Delays
Doostmohammadian, M., Aghasi, A., Rikos, A. I., Grammenos, A., Kalyvianaki, E., Hadjicostis, C. N., Johansson, K. H., & Charalambous, T. (2022) · IEEE Open Journal of Control Systems · doi:10.1109/ojcsys.2022.3210453 · Cited by 40
“The existing distributed optimization methods in both constrained [5] and unconstrained settings [6], [7], [8] are mostly linear.”
MethodLi, H., Lin, Z., & Fang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.04373 · Cited by 40
“To further improve the convergence of stochastic decentralized algorithms, Pu & Nedić (2021) combined DSGD with gradient tracking, Mokhtari & Ribeiro (2016) combined EXTRA with SAGA, and proposed the decentralized double stochastic averaging gradient algorithm, Xin et al. (2020b) combined gradient tracking with the V…”
BackgroundMethodInfluentialA Newton Tracking Algorithm With Exact Linear Convergence for Decentralized Consensus Optimization
Zhang, J., Ling, Q. H., & So, A. M. (2021) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2021.3083975 · Cited by 39
A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex OptimizationPreprint
Xin, R., Khan, U. A., & Kar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06752 · Cited by 38
“…the following quantities: In the following lemma, we enlist several well-known results in the context of gradient tracking-based algorithms for decentralized stochastic optimization, whose proofs may be found in (Di Lorenzo & Scutari, 2016; Qu & Li, 2017; Xin et al., 2020d; Pu & Nedich, 2020).”
BackgroundMethodInfluentialSelf-citationDecentralized Federated Learning With Intermediate Results in Mobile Edge Computing
Chen, S., Xu, Y., Xu, H., Jiang, Z., & Qiao, C. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3221212 · Cited by 33
“The works [37], [38], [39] tried to improve the convergence rate and accelerate training by reducing the gradient variance.”
BackgroundPrivacy-Preserving Distributed ADMM With Event-Triggered Communication
Zhang, Z., Yang, S., Xu, W., & Di, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3192346 · Cited by 32
“Typical first-order methods include gradient descent method [9]–[13], exact first-order method [14], gradient tracking algorithm [15], [16], exact diffusion [17], and linearized alternating direction method of multipliers (ADMM) [18].”
MethodMintoo, A. A., Saimon, A. S. M., Bakhsh, M. M., & Akter, M. (2022) · American Journal of Scholarly Research and Innovation · doi:10.63125/sdz8km60 · Cited by 30
Xin, R., Das, S., Khan, U. A., & Kar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01594 · Cited by 30
Self-citationLi, H., Hu, J., Ran, L., Wang, Z., Lü, Q., Du, Z., & Huang, T. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3072373 · Cited by 28
Decentralized Edge Learning via Unreliable Device-to-Device Communications
Jiang, Z., Yu, G., Cai, Y., & Jiang, Y. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3172147 · Cited by 28
Scaling-Up Distributed Processing of Data Streams for Machine Learning
Nokleby, M., Raja, H., & Bajwa, W. U. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3021381 · Cited by 27
“Several of the key findings of such works have also been elucidated through excellent survey articles and overview papers in recent years [14]–[23].”
BackgroundLi, B., Li, Z., & Chi, Y. (2022) · SIAM Journal on Mathematics of Data Science · doi:10.1137/21m1450677 · Cited by 26
Liu, H., Yu, W., Zheng, W. X., Nedić, A., & Zhu, Y. (2023) · Automatica · doi:10.1016/j.automatica.2023.111346 · Cited by 24
D-DistADMM: A O(1/k) Distributed ADMM for Distributed Optimization in Directed Graph Topologies
Khatana, V., & Salapaka, M. V. (2020) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc42340.2020.9304086 · Cited by 24
Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning
Zhang, J., Liu, H., So, A. M., & Ling, Q. H. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3240652 · Cited by 24
Perlaza, S. M., Bisson, G., Esnaola, I., Jean‐Marie, A., & Rini, S. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834273 · Cited by 23
A Computation-Efficient Decentralized Algorithm for Composite Constrained Optimization
Lü, Q., Liao, X., Li, H., & Huang, T. (2020) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2020.3037837 · Cited by 23
“Although several existing decentralized variance reduced methods such as DSA [45], GT-SAGA/GT-SVRG [51], etc., have been investigated for various kinds of problems, it is worth highlighting that via employing the variance reduction technique, Algorithm 1 is well applied to a class of composite optimization problems th…”
BackgroundMethodMomentum-based accelerated algorithm for distributed optimization under sector-bound nonlinearity
Doostmohammadian, M., & Rabiee, H. R. (2025) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2025.107857 · Cited by 22
Bayesian Variational Federated Learning and Unlearning in Decentralized Networks
Gong, J., Simeone, O., & Kang, J. (2021) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc51858.2021.9593225 · Cited by 22
A distributed algorithm based on relaxed ADMM for energy resources coordination
Liu, H., Shi, Y., Wang, Z., Ran, L., Lü, Q., & Li, H. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107482 · Cited by 22
Dual-Free Stochastic Decentralized Optimization with Variance ReductionPreprint
Hendrikx, H., Bach, F., & Massoulié, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14384 · Cited by 20
“Figure 1 compares the performance of DVR with that of state-of-the-art primal algorithms such as EXTRA [37], NIDS [21], GT-SAGA [41], and Catalyst accelerated versions of EXTRA [20] and DVR.”
MethodCompetitive search algorithm: a new method for stochastic optimization
Xu, Y., Liu, H., Xie, S., Xi, L., & Lu, M. (2022) · Applied Intelligence · doi:10.1007/s10489-021-03133-4 · Cited by 19
“Therefore, many scholars have improved the existing algorithms or proposed novel algorithms to solve the different problems in engineering applications [22, 23].”
BackgroundLogarithmically Quantized Distributed Optimization Over Dynamic Multi-Agent Networks
Doostmohammadian, M., & Pequito, S. (2024) · IEEE Control Systems Letters · doi:10.1109/lcsys.2024.3487796 · Cited by 17
“Motivating applications include networked empirical risk minimization [28], distributed logistic regression [29], distributed support vector machine (SVM) [30], and classification via neural networks [31].”
BackgroundUnified Analysis of Decentralized Gradient Descent: A Contraction Mapping Framework
Larsson, E. G., & Michelusi, N. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3557332 · Cited by 17
“Noteworthy are also the overview papers [17], [18], and the book [19], with tutorial-style expositions of different versions of DGD.”
Distributed Saddle Point Problems for Strongly Concave-Convex Functions
Qureshi, M. I., & Khan, U. A. (2023) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2023.3317807 · Cited by 17
“Of note are also distributed stochastic problems where gradient tracking is combined with variance reduction to achieve state-of-the-art results for several different classes of problems [39], [42]–[47].”
BackgroundSelf-citationAsynchronous Decentralized Optimization in Directed NetworksPreprint
Zhang, J., & You, K. (2019) · arXiv (Cornell University) · Cited by 17
Accelerated Distributed Stochastic Nonconvex Optimization Over Time-Varying Directed Networks
Chen, Y., Hashemi, A., & Vikalo, H. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3479888 · Cited by 16
Federated Learning Using Variance Reduced Stochastic Gradient for Probabilistically Activated Agents
Rostami, M., & Kia, S. S. (2023) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc55779.2023.10155945 · Cited by 15
“The randomness in SGD algorithms induces variance of the gradient, which leads to decay learning rate and sub-linear convergence rate.”
MethodInfluentialDecentralized Learning over Wireless Networks: The Effect of Broadcast with Random Access
Chen, Z., Dahl, M., & Larsson, E. G. (2023) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc53906.2023.10304514 · Cited by 14
“Consensus-based decentralized stochastic gradient descent (DSGD) is a widely adopted approach for collaborative training of machine learning models over networked agents, in which each agent combines local gradient computation with consensus-based model updating in an iterative manner [2], [3].”
MethodS-NEAR-DGD: A Flexible Distributed Stochastic Gradient Method for Inexact Communication
Iakovidou, C., & Wei, E. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3151734 · Cited by 13
“setting, existing approaches utilizing stochastic gradients include variations of DGD [21], [22], stochastic diffusion algorithms [23]–[25], primal-dual methods [20], [26]–[29], gradient-push algorithms [30], [31], the dual-averaging method [32], accelerated distributed algorithms [33], and stochastic distributed GT m…”
MethodHu, J., Chen, G., Li, H., Shen, Z., & Zhang, W. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3225229 · Cited by 13
The Effectiveness of Local Updates for Decentralized Learning Under Data Heterogeneity
Wu, T., Li, Z., & Sun, Y. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3533208 · Cited by 12
Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training
Ko, Y., Lee, D., & Kim, S. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511989 · Cited by 11
“To speed up such training, a large-batch training with data parallelism has been widely adopted [1, 2, 7, 8, 12, 14, 23, 32, 37, 38, 44–46].”
MethodVariance Reduction-Boosted Byzantine Robustness in Decentralized Stochastic Optimization
Peng, J., Li, W., & Ling, Q. H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746340 · Cited by 11
Variance Reduced Stochastic Optimization Over Directed Graphs with Row and Column Stochastic Weights
Qureshi, M. I., Xin, R., Kar, S., & Khan, U. A. (2023) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf59524.2023.10476854 · Cited by 11
“Distributed solutions are thus preferable where individual nodes perform local updates using data fusion among the nearby nodes [3–5].”
BackgroundSelf-citationHu, J., Ran, L., Du, Z., & Li, H. (2020) · Signal Processing · doi:10.1016/j.sigpro.2020.107894 · Cited by 11
ET-DASG: An Efficient Decentralized Algorithm for Convex Optimization Over Networks
Lü, Q., Liao, X., Deng, S., & Li, H. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3153386 · Cited by 10
SARAH-M: A fast stochastic recursive gradient descent algorithm via momentum
Yang, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122295 · Cited by 10
Hu, J., Xia, D., Cheng, H., Feng, L., Ji, L., Guo, J., & Li, H. (2020) · Asian Journal of Control · doi:10.1002/asjc.2483 · Cited by 10
“Although Xin et al. [11] and Mokhtari and Ribeiro [22] can achieve the exact convergence, the limitation of undirected or balanced directed networks may restrict the applicability of them.”
BackgroundMethodALADDIN: Asymmetric Centralized Training for Distributed Deep Learning
Ko, Y., Choi, K., Jei, H., Lee, D., & Kim, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482412 · Cited by 10
“Decentralized training [1, 2, 21, 22, 31] aggregates the training results of workers via peer-to-peer communication, avoiding the bottleneck problem.”
BackgroundDelay-tolerant augmented-consensus-based distributed directed optimization
Doostmohammadian, M., Narahari, K. R., & Aghasi, A. (2025) · Systems & Control Letters · doi:10.1016/j.sysconle.2025.106260 · Cited by 9
Communication Efficient Federated Learning via Ordered ADMM in a Fully Decentralized Setting
Chen, Y.-C., Blum, R. S., & Sadler, B. M. (2022) · Conference on Information Sciences and Systems · doi:10.1109/ciss53076.2022.9751166 · Cited by 9
Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning
Zhao, X., Lei, J., Li, L., Buşoniu, L., & Xu, J. (2025) · Journal of Systems Science and Complexity · doi:10.1007/s11424-025-4426-7 · Cited by 8
“After receiving the parameter θ s +1 j,k from its neighbors j ∈ N i , agent i updates θ s +1 i,k +1 via By using the dynamic average-consensus [45] , each agent refines the mix of neighboring gradient trackers y s +1 j,k with local gradient estimator v s +1 i,k +1 by where y 1 i, 0 = v 1 i, 0 , ∀ i…”
Decentralized Constrained Optimization: Double Averaging and Gradient Projection
Shahriari-Mehr, F., Bosch, D. D., & Panahi, A. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9683355 · Cited by 8
“Several classes of methods exist in the literature that are not in the scope of this paper, e.g. , methods considering time-varying graphs [19, 20], local functions with a finite-sum structure [8, 9, 17,33], or compressed communication [3,12].”
MethodA Decentralized Stochastic Algorithm for Coupled Composite Optimization With Linear Convergence
Lü, Q., Liao, X., Deng, S., & Li, H. (2022) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2022.3190743 · Cited by 8
FedNMUT -- Federated Noisy Model Update Tracking Convergence AnalysisPreprint
Chellapandi, V. P., Upadhyay, A., Hashemi, A., & Żak, S. H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.13247 · Cited by 8
“This rate of convergence is comparable to the best-known results for decentralized SGD algorithms found in existing literature [33], [34].”
ResultDoostmohammadian, M., Gabidullina, Z. R., & Rabiee, H. R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113018 · Cited by 7
Al-Kaabi, R. A., Fakhruldeen, H. F., & Al-Asady, H. A.-J. (2023) · AIP conference proceedings · doi:10.1063/5.0120150 · Cited by 7
Online Distributed Learning with Quantized Finite-Time Coordination
Bastianello, N., Rikos, A. I., & Johansson, K. H. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383824 · Cited by 7
“…decreases as t increases (as a consequence of the well known convergence of the average consensus). b) Gradient tracking: Rather than (approximately) projecting local gradient steps onto the consensus set, the extensively studied approach of gradient tracking can be used instead, see e.g. [15].”
BackgroundInfluentialDistributed Quantile Regression with Non-Convex Sparse Penalties
Mirzaeifard, R., Gogineni, V. C., Venkategowda, N. K. D., & Werner, S. (2023) · IEEE Statistical Signal Processing Workshop (SSP) · doi:10.1109/ssp53291.2023.10208080 · Cited by 7
“, [22]–[27], can be directly applied to the quantile regression context.”
BackgroundZhao, S., & Liu, Y. (2023) · Automatica · doi:10.1016/j.automatica.2023.111352 · Cited by 7
“…rate of iterates; [31] extends DGT to the distributed stochastic gradient tracking (DSGT) method and shows that the iterates generated by each agent converge to a neighborhood of the optimal solution in a linear rate; [46,33] propose the AB /push-pull method which applies the gradient…”
FedScalar: Federated Learning with Scalar Communication for Bandwidth-Constrained NetworksPreprint
Rostami, M. R., & Kia, S. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.02260 · Cited by 7
“Each agent receives the current model parameters from a central server and independently updates the model using its own dataset, as described in [6].”
Zhang, S., Pananjady, A., & Romberg, J. (2025) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2025.3562997 · Cited by 6
“The line of work [39], [40], [41], [42], [43], [44], [45], [46], [47] along these lines addresses problems where local objectives have finite-sum structure.”
Consensus-Based Distributed Computation of Link-Based Network Metrics
Chen, Z., & Larsson, E. G. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3050056 · Cited by 6
“CONSENSUS mechanism is the foundation of distributed systems in the absence of centralized management, which is a long-standing research direction with promising applications in diverse fields, such as swarm robotics [1], opinion dynamics in social networks [2], and emerging decentralized machine learning applications…”
BackgroundDecentralized Stochastic Variance Reduced Extragradient MethodPreprint
Luo, L., & Ye, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.00509 · Cited by 6
“[43] Ran Xin, Soummya Kar, and Usman A Khan.”
BackgroundLi, X., Xu, Y., Wang, J. H., Wang, X., & Lui, J. C. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10389 · Cited by 6
“Owing to the above considerations, variance reduction is generally used in decentralized learning, including DSA [17], NetworkSVRG/SARAH [18] and GT-SVRG [19].”
BackgroundAsynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning
Chen, H., Ye, Y., Xiao, M., & Skoglund, M. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3230335 · Cited by 5
“Related works: With the rapid popularity of decentralized ML, for instance, federated learning coined by Google in 2016 [6] , it has become an attractive research direction in recent years [7], [8], [9], [10], [11].”
BackgroundPeng, J., Li, W., & Ling, Q. H. (2024) · Signal Processing · doi:10.1016/j.sigpro.2024.109419 · Cited by 5
A Decentralized Variance-Reduced Method for Stochastic Optimization Over Directed Graphs
Qureshi, M. I., Xin, R., Kar, S., & Khan, U. A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413600 · Cited by 5
Self-citationPeer-to-Peer Learning+Consensus with Non-IID Data
Pranav, S., & Moura, J. M. F. (2023) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf59524.2023.10476902 · Cited by 5
“Gradient tracking methods [13] were developed for distributed convex optimization problems where local objective functions do not share the same minimum as the global objective.”
MethodDC-DistADMM: ADMM Algorithm for Contsrained Distributed Optimization over Directed GraphsPreprint
Khatana, V., & Salapaka, M. V. (2020) · arXiv (Cornell University) · Cited by 5
Decentralized Federated Learning: Model Update Tracking Under Imperfect Information Sharing
Chellapandi, V. P., Upadhyay, A., Hashemi, A., & Żak, S. H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825274 · Cited by 4
Ran, L., Li, H., Hu, J., Lü, Q., Wang, Z., Li, Z., & Chen, G. (2024) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2023.3285865 · Cited by 4
“By contrast, distributed optimization methods develop a brand-new perspective for alleviating the computation cost in various engineering applications, including machine learning [17], wireless sensor networks [18], and resource allocation problems [19], etc.”
BackgroundShahriari-Mehr, F., & Panahi, A. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3520513 · Cited by 4
“For a comprehensive survey, see [7], [8], [9], and [10].”
DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction
Pan, H., & Zheng, L. (2021) · Sensors · doi:10.3390/s21155124 · Cited by 4
“We found that the DisSAGD algorithm performance depends on the specific dataset, i.e., it does not exceed SSGD on KDDcup99, whereas it outperforms SSGD on Cifar100, but the overall convergence performance is better than the performance of ASD-SVRG, which significantly better than that of PetuumSGD and SSGD, and is clo…”
MethodInfluentialDoostmohammadian, M., Aghasi, A., & Rabiee, H. R. (2024) · RSI International Conference on Robotics and Mechatronics (ICRoM) · doi:10.1109/icrom64545.2024.10903613 · Cited by 3
“Emerging notions of cloud-computing and multi-agent processing motivate distributed and parallelized algorithms in large-scale applications from filtering and estimation over sensor networks [1]–[3] to distributed machine learning [4]– [6] and computing resource allocation [7], [8].”
GT-SAGA: A fast incremental gradient method for decentralized finite-sum minimization
Xin, R., Li, B., Kar, S., & Khan, U. A. (2020) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc42340.2020.9304007 · Cited by 3
“It can be shown that DSGD achieves (cid:15) -accuracy of the solution in O ( − 1 ) gradient computations and certain aspects of it are improved with the addition of gradient tracking, see [4] for a detailed tutorial.”
BackgroundMethodSelf-citationDistributed Stochastic Learning for Composite Sharing Optimization in Consumer Electronics
Lü, Q., Dai, X., Zhang, W., & Ezilarasan, M. R. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3506915 · Cited by 3
“To reduce the variance in stochastic gradients and achieve exact convergence, variance-reduction techniques are adopted in centralized approaches [10], [41] and subsequently in distributed approaches [42], [43] to overcome this drawback.”
Xiong, X.-C., Zhang, H.-Z., Peng, G., & Feng, F. (2025) · Journal of Asian Architecture and Building Engineering · doi:10.1080/13467581.2025.2457380 · Cited by 3
“Currently, widely used stochastic gradient descent and its enhancements perform exceptionally well in structural optimization (Haji and Abdulazeez 2021; Meng et al. 2021; F. Wang et al. 2023; Xin, Kar, and Khan 2020).”
DIN: A Decentralized Inexact Newton Algorithm for Consensus Optimization
Ghalkha, A., Issaid, C. B., Elgabli, A., & Bennis, M. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3400756 · Cited by 3
Gramian-Based Adaptive Combination Policies for Diffusion Learning Over Networks
Erginbas, Y. E., Vlaski, S., & Sayed, A. H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414449 · Cited by 3
Decentralized Dual Operator Splitting for Nonsmooth Composite Optimization
Li, H., Ding, W., Wang, Z., Lü, Q., Li, Y., & Xia, D. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3155287 · Cited by 3
Metadata
- Authors
- Xin, R., Kar, S., & Khan, U. A.
- Affiliations
- Carnegie Mellon University (US)
- Tufts University (US)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2020-05
- Volume
- 37
- Issue
- 3
- Pages
- 102-113
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3023944502
- Cited by
- 114
- References
- 53
- Citations by year
- 2026: 5 · 2025: 15 · 2024: 13 · 2023: 16 · 2022: 28 · 2021: 22 · 2020: 13
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Funders
- National Science Foundation (CCF-1350264; CCF-1513936; CMMI-1903972; CBET-1935555)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- distributed optimization, stochastic optimization, variance reduction, gradient tracking, finite-sum minimization, decentralized machine learning, decentralized learning, non-convex optimization, strong convexity, convergence guarantees, privacy-preserving machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.