The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Decentralized Federated Averaging
Open accessAbstract
Federated averaging (FedAvg) is a communication-efficient algorithm for distributed training with an enormous number of clients. In FedAvg, clients keep their data locally for privacy protection; a central parameter server is used to communicate between clients. This central server distributes the parameters to each client and collects the updated parameters from clients. FedAvg is mostly studied in centralized fashions, requiring massive communications between the central server and clients, which leads to possible channel blocking. Moreover, attacking the central server can break the whole system's privacy. Indeed, decentralization can significantly reduce the communication of the busiest node (the central one) because all nodes only communicate with their neighbors. To this end, in this paper, we study the decentralized FedAvg with momentum (DFedAvgM), implemented on clients that are connected by an undirected graph. In DFedAvgM, all clients perform stochastic gradient descent with momentum and communicate with their neighbors only. To further reduce the communication cost, we also consider the quantized DFedAvgM. The proposed algorithm involves the mixing matrix, momentum, client training with multiple local iterations, and quantization, introducing extra items in the Lyapunov analysis. Thus, the analysis of this paper is much more challenging than previous decentralized (momentum) SGD or FedAvg. We prove convergence of the (quantized) DFedAvgM under trivial assumptions; the convergence rate can be improved to sublinear when the loss function satisfies the PŁ property. Numerically, we find that the proposed algorithm outperforms FedAvg in both convergence speed and communication cost.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Consensus and Cooperation in Networked Multi-Agent Systems
Olfati-Saber, R., Fax, J. A., & Murray, R. M. (2007) · Proceedings of the IEEE · doi:10.1109/jproc.2006.887293 · Cited by 10,522 · cited 1×
“Decentralized algorithmswere originally developed to calculate the mean of data that are stored over multiple sensors [21], [22], [23], [24].”
BackgroundA Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
(2013) · Cited by 3,740
Distributed Subgradient Methods for Multi-Agent Optimization
Nedic, A., & Ozdaglar, A. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.2009515 · Cited by 3,719
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 3,069
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
A survey on federated learning
Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., & Gao, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106775 · Cited by 1,899
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · arXiv · doi:10.48550/arxiv.1910.06378 · Cited by 1,734
Expander graphs and their applications
Hoory, S., Linial, N., & Wigderson, A. (2006) · Bulletin of the American Mathematical Society · doi:10.1090/s0273-0979-06-01126-8 · Cited by 1,723
Pegasos: primal estimated sub-gradient solver for SVM
Shalev-Shwartz, S., Singer, Y., Srebro, N., & Cotter, A. (2011) · Mathematical Programming · doi:10.1007/s10107-010-0420-4 · Cited by 1,477
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
Ghadimi, S., & Lan, G. (2013) · SIAM Journal on Optimization · doi:10.1137/120880811 · Cited by 1,034
Lectures on Convex Optimization
Nesterov, Y. (2018) · Springer Optimization and Its Applications · doi:10.1007/978-3-319-91578-4 · Cited by 1,010
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 859
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23119 · Cited by 852
Adaptive Federated Optimization
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., & McMahan, H. B. (2020) · arXiv · doi:10.48550/arxiv.2003.00295 · Cited by 840
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition
Karimi, H., Nutini, J., & Schmidt, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46128-1_50 · Cited by 824
A Survey on Federated Learning for Resource-Constrained IoT Devices
Imteaj, A., Thakker, U., Wang, S., Li, J., & Amini, M. H. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3095077 · Cited by 774
Fastest Mixing Markov Chain on a Graph
Boyd, S., Diaconis, P., & Xiao, L. (2004) · SIAM Review · doi:10.1137/s0036144503423264 · Cited by 744
On the Convergence of Decentralized Gradient Descent
Yuan, K., Ling, Q., & Yin, W. (2013) · arXiv · doi:10.48550/arxiv.1310.7063 · Cited by 635
On the Convergence of Decentralized Gradient Descent
Yuan, K., Ling, Q., & Yin, W. (2016) · SIAM Journal on Optimization · doi:10.1137/130943170 · Cited by 633
Fast Distributed Gradient Methods
Jakovetic, D., Xavier, J., & Moura, J. M. F. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2298712 · Cited by 624
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
Hsu, T.-M. H., Qi, H., & Brown, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06335 · Cited by 610
Gossip algorithms: design, analysis and applications
Boyd, S., Ghosh, A., Prabbakar, B., & Shah, D. (2005) · Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies. · doi:10.1109/infcom.2005.1498447 · Cited by 558
Yu, H., Yang, S., & Zhu, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015693 · Cited by 497
Broadcast Gossip Algorithms for Consensus
Aysal, T. C., Yildiz, M. E., Sarwate, A. D., & Scaglione, A. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2016247 · Cited by 480
Communication Efficient Distributed Machine Learning with the Parameter Server
Li, M., Andersen, D. G., Smola, A. J., & Yu, K. (2014) · neural information processing systems · Cited by 475
Lian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W., & Liu, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09056 · Cited by 395
Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T., & Yu, H. (2019) · Synthesis lectures on artificial intelligence and machine learning · doi:10.2200/s00960ed2v01y201910aim043 · Cited by 261
On the Convergence of Federated Optimization in Heterogeneous Networks.Preprint
Sahu, A. K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., & Smith, V. (2018) · arXiv (Cornell University) · Cited by 237
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
Liu, C., Zhu, L., & Belkin, M. (2022) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2021.12.009 · Cited by 234
A distributed consensus protocol for clock synchronization in wireless sensor network
Schenato, L., & Gamba, G. (2007) · 2007 46th IEEE Conference on Decision and Control · doi:10.1109/cdc.2007.4434671 · Cited by 224
Xin, R., & Khan, U. A. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2942513 · Cited by 201
Communication-efficient algorithms for decentralized and stochastic optimization
Lan, G., Lee, S., & Zhou, Y. (2020) · Mathematical Programming · doi:10.1007/s10107-018-1355-4 · Cited by 187
On Nonconvex Decentralized Gradient Descent
Zeng, J., & Yin, W. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2818081 · Cited by 184
FedDANE: A Federated Newton-Type Method
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smithy, V. (2019) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf44664.2019.9049023 · Cited by 159
A fast distributed proximal-gradient method
Chen, A. I., & Ozdaglar, A. (2012) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2012.6483273 · Cited by 154
Peer-to-peer Federated Learning on Graphs
Lalitha, A., Kilinc, O. C., Javidi, T., & Koushanfar, F. (2019) · arXiv · doi:10.48550/arxiv.1901.11173 · Cited by 146
Matei, I., & Baras, J. S. (2011) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2011.2120593 · Cited by 133
Decentralized Federated Learning via SGD over Wireless D2D Networks
Xing, H., Simeone, O., & Bi, S. (2020) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc48557.2020.9154332 · Cited by 122
SGD and Hogwild! Convergence Without the Bounded Gradients AssumptionPreprint
Nguyen, L. M., Nguyen, P. H., van Dijk, M., Richtárik, P., Scheinberg, K., & Takáč, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03801 · Cited by 118
First Analysis of Local GD on Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04715 · Cited by 103
Communication-Efficient Distributed Learning via Lazily Aggregated Quantized GradientsPreprint
Sun, J., Chen, T., Giannakis, G. B., & Yang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07588 · Cited by 83
Magnússon, S., Shokri‐Ghadikolaei, H., & Li, N. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3031073 · Cited by 77
Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms
Sun, T., Yin, P., Li, D., Huang, C., Guan, L., & Jiang, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015033 · Cited by 44
Self-citationConsensus optimization with delayed and stochastic gradients on decentralized networks
Sirb, B., & Ye, X. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840591 · Cited by 29
Quantized Decentralized Consensus Optimization
Reisizadeh, A., Mokhtari, A., Hassani, H., & Pedarsani, R. (2018) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2018.8619539 · Cited by 29
(2020) · International Conference on Learning Representations
Cited by
Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges
Martínez Beltrán, E. T., Pérez, M. Q., Sánchez, P. M. S., Bernal, S. L., Bovet, G., Pérez, M. G., Pérez, G. M., & Celdrán, A. H. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3315746 · Cited by 625
“better convergence rate and communication efficiency than DSGD [129].”
BackgroundAbbas, S. R., Abbas, Z., Zahir, A., & Lee, S. W. (2024) · Healthcare · doi:10.3390/healthcare12242587 · Cited by 228
A collective AI via lifelong learning and sharing at the edge
Soltoggio, A., Ben-Iwhiwhu, E., Braverman, V., Eaton, E., Epstein, B., Ge, Y., Halperin, L., How, J., Itti, L., Jacobs, M. A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00800-2 · Cited by 51
Papadopoulos, C., Kollias, K.-F., & Fragulis, G. F. (2024) · Future Internet · doi:10.3390/fi16110415 · Cited by 35
Privacy Preserving Federated Learning for Student Dropout Prediction Using Sensor and EMA Data
Nadim, M. A., Marsico, E., & Di Fuccio, R. (2025) · 2025 3rd International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391036
Metadata
- Authors
- Sun, T., Li, D., & Wang, B.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-04-01
- Volume
- 45
- Issue
- 4
- Pages
- 4289-4301
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3158567365
- Cited by
- 388
- References
- 52
- FWCI
- 24.88
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.