The DOI matches its DataCite record; 42 of the 45 references it cites were found in a registry or an index.
Adaptive Federated Optimization
Open accessAbstract
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“MPC approaches to PIR break down into two main categories: computational PIR (cPIR), in which a single party can execute the entire server side of the protocol [286], and information theoretic PIR (itPIR), in which multiple non-colluding parties are required to execute the server side of the protocol [121].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [121]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Self-citation(2010) · Cited by 8,261
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Self-citationFederated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Self-citationWu, Y., & He, K. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01198-w · Cited by 3,601
EMNIST: Extending MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J. C., & André, V. S. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966217 · Cited by 2,410
Towards Federated Learning at Scale: System Design
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H. B., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.01046 · Cited by 2,334
Self-citationAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
On the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
Self-citationMeasuring the Effects of Non-Identical Data Distribution for Federated Visual ClassificationPreprint
Hsu, T., Qi, & Brown, M. (2019) · arXiv.org · doi:10.48550/arxiv.1909.06335 · Cited by 1,757
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · arXiv · doi:10.48550/arxiv.1910.06378 · Cited by 1,734
Self-citationOn the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
One weird trick for parallelizing convolutional neural networksPreprint
Krizhevsky, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.5997 · Cited by 1,372
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
LEAF: A Benchmark for Federated Settings
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.01097 · Cited by 894
Self-citationYu, H., Yang, S., & Zhu, S.-H. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015693 · Cited by 696
Local SGD Converges Fast and Communicates LittlePreprint
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 679
Li, W., & McCallum, A. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143917 · Cited by 627
Lookahead Optimizer: k steps forward, 1 step backPreprint
Zhang, M. R., Lucas, J., Hinton, G., & Ba, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08610 · Cited by 503
Stochastic Variance Reduction for Nonconvex OptimizationPreprint
Reddi, S. J., Hefny, A., Sra, S., Póczos, B., & Smola, A. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06160 · Cited by 415
Self-citationSAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite ObjectivesPreprint
Defazio, A., Bach, F., & Lacoste-Julien, S. (2014) · arXiv (Cornell University) · Cited by 372
How to Escape Saddle Points EfficientlyPreprint
Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., & Jordan, M. I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00887 · Cited by 360
Adaptive Gradient Methods with Dynamic Bound of Learning RatePreprint
Luo, L., Xiong, Y., Liu, Y., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09843 · Cited by 353
The Non-IID Data Quagmire of Decentralized Machine LearningPreprint
Hsieh, K., Phanishayee, A., Mutlu, O., & Gibbons, P. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00189 · Cited by 353
Wang, J., & Joshi, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07576 · Cited by 302
Adaptive Methods for Nonconvex Optimization
Zaheer, M., Reddi, S. J., Sachan, D. S., Kale, S., & Kumar, S. (2018) · neural information processing systems · Cited by 267
Self-citationSCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · Cited by 227
Self-citationAdaptive bound optimization for online convex optimization
McMahan, H. B., & Streeter, M. (2010) · arXiv · doi:10.48550/arxiv.1002.4908 · Cited by 227
Self-citationFirst Analysis of Local GD on Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04715 · Cited by 103
Stich, S. U., & Karimireddy, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05350 · Cited by 91
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initializationPreprint
Ward, R., Wu, X., & Bottou, L. (2018) · arXiv (Cornell University) · Cited by 59
Why ADAM Beats SGD for Attention Models
Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S. J., Kumar, S., & Sra, S. (2019) · Cited by 54
Self-citationA Generic Approach for Escaping Saddle pointsPreprint
Reddi, S. J., Zaheer, M., Sra, S., Póczos, B., Bach, F., Salakhutdinov, R., & Smola, A. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.01434 · Cited by 54
Self-citationEscaping Saddle Points with Adaptive Gradient MethodsPreprint
Staib, M., Reddi, S. J., Kale, S., Kumar, S., & Sra, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09149 · Cited by 49
Self-citationGlobal Convergence of Adaptive Gradient Methods for An Over-parameterized Neural NetworkPreprint
Wu, X., Du, S. S., & Ward, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07111 · Cited by 37
On the Convergence of Stochastic Gradient Descent with Adaptive StepsizesPreprint
Li, X., & Orabona, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08114 · Cited by 32
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning RatesPreprint
Xie, C., Koyejo, S., Gupta, I., & Lin, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09030 · Cited by 21
Flagged first, then by citations
3 of the 45 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-SGD: Distributed SGD with quantization, sparsification and local computations. In Advances in Neural Information Processing Systems , pp. 14668-14679, 2019.
- Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. arXiv preprint arXiv:1812.04529 , 2018. · arXiv
- H. Brendan McMahan and Matthew J. Streeter. Adaptive bound optimization for online convex optimization. In COLT The 23rd Conference on Learning Theory , 2010b.
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Self-citationA Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407
Federated Learning on Non-IID Data Silos: An Experimental Study
Li, Q., Diao, Y., Chen, Q., & He, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00077 · Cited by 1,407
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2007.07481 · Cited by 927
Robust Aggregation for Federated Learning
Pillutla, K., Kakade, S. M., & Harchaoui, Z. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3153135 · Cited by 1,068
Metadata
- Authors
- Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., & McMahan, H. B.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.00295
- OpenAlex
- W3008187686
- Cited by
- 840
- References
- 45
- Citations by year
- 2026: 10 · 2025: 127 · 2024: 233 · 2023: 222 · 2022: 120 · 2021: 92 · 2020: 35
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- federated learning, adaptive optimization, FedAvg, AdaGrad, Adam, yogis, convergence analysis, non-convex optimization, client heterogeneity, communication efficiency, distributed learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
