The DOI matches its DataCite record; 37 of the 40 references it cites were found in a registry or an index.
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Open accessAbstract
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized message-passing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“by optimizing the workload assignment weights and computing capacities of VMs, MASM [173] can reduce the energy cost and delay while guaranteeing the DL inference accuracy.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [173]
Citations
Cites
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
On First-Order Meta-Learning Algorithms
Nichol, A., Achiam, J., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1803.02999 · Cited by 1,438
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Alistarh, D., Grubic, D., Li, J. Z., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 1,262
The Tradeoffs of Large-Scale Learning
Bottou, L., & Bousquet, O. (2011) · The MIT Press eBooks · doi:10.7551/mitpress/8996.003.0015 · Cited by 1,201
Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
Yin, D., Chen, Y., Ramchandran, K., & Bartlett, P. (2018) · arXiv · doi:10.48550/arxiv.1803.01498 · Cited by 1,130
Federated Learning of Deep Networks using Model AveragingPreprint
McMahan, H. B., Moore, E., Ramage, D., & y Arcas, B. A. (2016) · arXiv (Cornell University) · Cited by 1,005
Learning Differentially Private Recurrent Language Models
McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017) · arXiv · doi:10.48550/arxiv.1710.06963 · Cited by 849
Speeding Up Distributed Machine Learning Using Codes
Lee, K., Lam, M., Pedarsani, R., Papailiopoulos, D., & Ramchandran, K. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2736066 · Cited by 829
Self-citationLocal SGD Converges Fast and Communicates LittlePreprint
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 679
Reptile: a Scalable Metalearning AlgorithmPreprint
Nichol, A., & Schulman, J. (2018) · arXiv (Cornell University) · Cited by 648
Distributed Federated Learning for Ultra-Reliable Low-Latency Vehicular Communications
Samarakoon, S., Bennis, M., Saad, W., & Debbah, M. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2956472 · Cited by 459
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443
Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix MultiplicationPreprint
Yu, Q., Maddah-Ali, M. A., & Avestimehr, A. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10464 · Cited by 439
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed CommunicationPreprint
Koloskova, A., Stich, S. U., & Jaggi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00340 · Cited by 377
cpSGD: Communication-efficient and differentially-private distributed SGDPreprint
Agarwal, N., Suresh, A. T., Yu, F., Kumar, S., & McMahan, H. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10559 · Cited by 373
“Short-Dot”: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products
Dutta, S., Cadambe, V. R., & Grover, P. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2019.2927558 · Cited by 361
Don't Use Large Mini-Batches, Use Local SGDPreprint
Lin, T., Stich, S. U., Patel, K. K., & Jaggi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07217 · Cited by 324
Wang, J., & Joshi, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07576 · Cited by 302
Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2014) · Journal of the ACM · doi:10.1145/2666468 · Cited by 254
The landscape of empirical risk for nonconvex losses
Mei, S., Bai, Y. D., & Andrea Montanari, A. (2018) · The Annals of Statistics · doi:10.1214/17-aos1637 · Cited by 250
On the Convergence of Federated Optimization in Heterogeneous Networks.Preprint
Sahu, A. K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., & Smith, V. (2018) · arXiv (Cornell University) · Cited by 237
Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
Chen, Y., Su, L., & Xu, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219655 · Cited by 228
Robust Federated Learning in a Heterogeneous EnvironmentPreprint
Ghosh, A., Hong, J., Yin, D., & Ramchandran, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06629 · Cited by 204
CoCoA: A General Framework for Communication-Efficient Distributed OptimizationPreprint
Smith, V., Forte, S., Ma, C., Takáč, M., Jordan, M. I., & Jaggi, M. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000282738 · Cited by 204
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
Wang, J., Sahu, A. K., Yang, Z., Joshi, G., & Kar, S. (2019) · 2019 Sixth Indian Control Conference (ICC) · doi:10.1109/icc47138.2019.9123209 · Cited by 149
Peer-to-peer Federated Learning on Graphs
Lalitha, A., Kilinc, O. C., Javidi, T., & Koushanfar, F. (2019) · arXiv · doi:10.48550/arxiv.1901.11173 · Cited by 146
An Exact Quantized Decentralized Gradient Descent Algorithm
Reisizadeh, A., Mokhtari, A., Hassani, H., & Pedarsani, R. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2932876 · Cited by 138
Self-citationOne-Shot Federated LearningPreprint
Guha, N., Talwalkar, A., & Smith, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11175 · Cited by 137
Balancing Communication and Computation in Distributed Optimization
Berahas, A. S., Bollapragada, R., Keskar, N. S., & Wei, E. (2018) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2018.2880407 · Cited by 108
Federated Heavy Hitters Discovery with Differential PrivacyPreprint
Zhu, W., Kairouz, P., McMahan, H. B., Sun, H., & Li, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08534 · Cited by 66
LoAdaBoost:Loss-Based AdaBoost Federated Machine Learning on medical DataPreprint
Li, H., Yin, Y., Fu, Z., Zhang, S., Deng, H., & Liu, D. (2018) · arXiv (Cornell University) · Cited by 57
CodedReduce: A Fast and Robust Framework for Gradient Aggregation in Distributed Learning
Reisizadeh, A., Prakash, S., Pedarsani, R., & Avestimehr, A. S. (2021) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2021.3109097 · Cited by 30
Self-citationCompressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks
Zhang, X., Liu, J., Zhu, Z., & Bentley, E. S. (2019) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2019.8737489 · Cited by 26
Accelerating the Convergence Rates of Distributed Subgradient Methods with Adaptive QuantizationPreprint
Doan, T. T., Maguluri, S. T., & Romberg, J. (2018) · arXiv (Cornell University) · Cited by 8
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
“FedPAQ [204] simultaneously incorporates these features and provides near-optimal theoretical guarantees for both strongly convex and non-convex loss functions, while empirically demonstrating the communication-computation tradeoff.”
BackgroundFederated Learning for Internet of Things: A Comprehensive Survey
Nguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., Li, J., & Vincent Poor, H. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3075439 · Cited by 1,503
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2007.07481 · Cited by 927
Metadata
- Authors
- Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., & Pedarsani, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 2021-2031
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.13014
- OpenAlex
- W2975043678
- Cited by
- 587
- References
- 40
- Citations by year
- 2025: 41 · 2024: 146 · 2023: 156 · 2022: 96 · 2021: 100 · 2020: 46 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Artificial Intelligence and Statistics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
- Keywords
- federated learning, communication efficiency, scalability, computation-communication trade-off, strongly convex loss, non-convex loss
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
