On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
Open accessAbstract
Recent developments on large-scale distributed machine learning applications, e.g., deep neural networks, benefit enormously from the advances in distributed non-convex optimization techniques, e.g., distributed Stochastic Gradient Descent (SGD). A series of recent works study the linear speedup property of distributed SGD variants with reduced communication. The linear speedup property enable us to scale out the computing capability by adding more computing nodes into our system. The reduced communication complexity is desirable since communication overhead is often the performance bottleneck in distributed systems. Recently, momentum methods are more and more widely adopted in training machine learning models and can often converge faster and generalize better. For example, many practitioners use distributed SGD with momentum to train deep neural networks with big data. However, it remains unclear whether any distributed momentum SGD possesses the same linear speedup property as distributed SGD and has reduced communication complexity. This paper fills the gap by considering a distributed communication efficient momentum SGD method and proving its linear speedup property.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most homomorphic encryption schemes require that secret keys be renewed often due to susceptibility to chosen ciphertext attacks.
“However, most HE schemes require that the secret keys be renewed often (due to e.g. susceptibility to chosen ciphertext attacks [117]).”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [117]
The cited paper discusses communication-efficient momentum SGD, including algorithm restarts in §6.3.1 (p. 13) and convergence analysis in §6.6 (p. 29). Searches for homomorphic encryption, secret-key renewal, and chosen-ciphertext attacks found no passage addressing the claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 37
- Aji, A. F. and Heafield, K. Sparse communication for distributed gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2017.
- Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Assran, M., Loizou, N., Ballas, N., and Rabbat, M. Stochastic gradient push for distributed deep learning. arXiv:1811.10792 , 2018. · arXiv
- Chen, K. and Huo, Q. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2016.
- Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(165-202), 2012.
- Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition (CVPR) , 2009.
- Ghadimi, E., Feyzmahdavian, H. R., and Johansson, M. Global convergence of the heavy-ball method for convex optimization. arXiv:1412.7457 , 2014. · arXiv
- Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.
- Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. arXiv:1706.02677 , 2017. · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR) , 2016.
- Horn, R. A. and Johnson, C. R. Matrix Analysis . Cambridge University Press, 1985.
- Jiang, P. and Agrawal, G. A linear speedup analysis of distributed deep learning with sparse and quantized communication. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.
- Jiang, Z., Balu, A., Hegde, C., and Sarkar, S. Collaborative deep learning in fixed topology networks. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Li, M., Andersen, D. G., Smola, A. J., and Yu, K. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems (NIPS) , 2014.
- Lian, X., Huang, Y., Li, Y., and Liu, J. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Information Processing Systems (NIPS) , 2015.
- Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Lin, T., Stich, S. U., and Jaggi, M. Don't use large minibatches, use local SGD. arXiv:1808.07217 , 2018. · arXiv
- Mania, H., Pan, X., Papailiopoulos, D., Recht, B., Ramchandran, K., and Jordan, M. I. Perturbed iterate analysis for asynchronous stochastic optimization. SIAM Journal on Optimization , 27(4):2202-2229, 2017.
- McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.
- Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course . Springer Science & Business Media, 2004.
- Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.
- Seide, F. and Agarwal, A. CNTK: Microsoft's open-source deep-learning toolkit. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 2016.
- Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochastic gradient descent and its application to dataparallel distributed training of speech DNNs. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2014.
- Stich, S. U. Local SGD converges fast and communicates little. arXiv:1805.09767 , 2018. · arXiv
- Strom, N. Scalable distributed DNN training using commodity GPU cloud computing. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2015.
- Su, H., Chen, H., and Xu, H. Experiments on parallel training of deep neural network using model averaging. arXiv:1507.01239v3 , 2018. · arXiv
- Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning (ICML) , 2013.
- Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. D 2 : Decentralized training over decentralized data. arXiv:1803.07068 , 2018. · arXiv
- Wang, J. and Joshi, G. Cooperative SGD: A unified framework for the design and analysis of communicationefficient SGD algorithms. arXiv:1808.07576 , 2018a. · arXiv
- Wang, J. and Joshi, G. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD. arXiv:1810.08313 , 2018b. · arXiv
- Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. A unified analysis of stochastic momentum methods for deep learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
- Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. arXiv:1807.06629 , 2018. · arXiv
- Zavriev, S. and Kostyuk, F. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling , 4(4):336-341, 1993.
- Zhou, F. and Cong, G. On the convergence properties of a K -step averaging stochastic gradient descent algorithm for nonconvex optimization. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · see citing works
Self-citationFederated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023 · see citing works
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2007.07481 · Cited by 927 · see citing works
Personalized Federated Learning with Moreau EnvelopesPreprint
Dinh, C. T., Tran, N. H. K., & Nguyen, T. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08848 · Cited by 644 · see citing works
Model Pruning Enables Efficient Federated Learning on Edge Devices
Jiang, Y., Wang, S., Valls, V., Ko, B. J., Lee, W., Leung, K. K., & Tassiulas, L. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3166101 · Cited by 594 · see citing works
Privacy-Preserving Federated Brain Tumour Segmentation
Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M. J., et al. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32692-0_16 · Cited by 578 · see citing works
Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks
Abad, M. S. H., Ozfatura, E., GUndUz, D., & Ercetin, O. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054634 · Cited by 424 · see citing works
Accelerating Federated Learning via Momentum Gradient Descent
Liu, W., Chen, L., Chen, Y., & Zhang, W. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.2975189 · Cited by 374 · see citing works
Fast-Convergent Federated Learning
Nguyen, H. T., Sehwag, V., Hosseinalipour, S., Brinton, C. G., Chiang, M., & Poor, H. V. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036952 · Cited by 281 · see citing works
Qsparse-local-SGD: Distributed SGD with quantization, sparsification, and local computations
Basu, D., Data, D., Karakus, C., & Diggavi, S. N. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2985917 · Cited by 256 · see citing works
FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data
Zhang, X., Hong, M., Dhople, S. V., Yin, W., & Liu, Y. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3115952 · Cited by 235 · see citing works
Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling
Luo, B., Xiao, W., Wang, S., Huang, J., & Tassiulas, L. (2022) · IEEE INFOCOM 2022 - IEEE Conference on Computer Communications · doi:10.1109/infocom48880.2022.9796935 · Cited by 229 · see citing works
Li, L., Shi, D., Hou, R.-H., Li, H., Pan, M., & Han, Z. (2020) · IEEE Conference on Computer Communications · doi:10.1109/infocom42981.2021.9488839 · Cited by 198 · see citing works
Federated Learning with Buffered Asynchronous AggregationPreprint
Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Esmaeili, M. M., & Huba, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06639 · Cited by 182 · see citing works
A Field Guide to Federated OptimizationPreprint
Wang, J., Charles, Z., Xu, Z., Joshi, G., McMahan, H. B., y Arcas, B. A., Al-Shedivat, M., Andrew, G., Avestimehr, S., Daly, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06917 · Cited by 181 · see citing works
Hu, S., Chen, X., Ni, W., Hossain, E., & Wang, X. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3086014 · Cited by 178 · see citing works
On the Convergence of Local Descent Methods in Federated LearningPreprint
Haddadpour, F., & Mahdavi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14425 · Cited by 176 · see citing works
Handling privacy-sensitive medical data with federated learning: Challenges and future directions
Aouedi, O., Sacco, A., Piamrat, K., & Marchetto, G. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3185673 · Cited by 165 · see citing works
Tighter Theory for Local SGD on Identical and Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04746 · Cited by 162 · see citing works
Decentralized Federated Learning: Balancing Communication and Computing Costs
Liu, W., Chen, L., & Zhang, W. (2022) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2022.3151242 · Cited by 163 · see citing works
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated LearningPreprint
Yang, H., Fang, M., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11203 · Cited by 150 · see citing works
An Improved Analysis of Stochastic Gradient Descent with MomentumPreprint
Liu, Y., Gao, Y., & Yin, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07989 · Cited by 137 · see citing works
Decentralized Deep Learning with Arbitrary Communication CompressionPreprint
Koloskova, A., Lin, T., Stich, S. U., & Jaggi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09356 · Cited by 118 · see citing works
Federated Accelerated Stochastic Gradient DescentPreprint
Yuan, H., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08950 · Cited by 106 · see citing works
First Analysis of Local GD on Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04715 · Cited by 103 · see citing works
Semisupervised Distributed Learning With Non-IID Data for AIoT Service Platform
Chiu, T.-C., Shih, Y.-Y., Pang, A.-C., Wang, C.-S., Weng, W., & Chou, C.-T. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2995162 · Cited by 98 · see citing works
Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering
Xu, J., Huang, S., Song, L., & Lan, T. (2022) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs54860.2022.00120 · Cited by 98 · see citing works
Faster Adaptive Federated Learning
Wu, X., Huang, F., Hu, Z., & Huang, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26235 · Cited by 96 · see citing works
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2020) · arXiv · doi:10.48550/arxiv.2008.03606 · Cited by 96 · see citing works
Communication-Efficient Distributed Deep Learning: A Comprehensive SurveyPreprint
Tang, Z.-H., Shi, S.-H., Wang, W., Li, B., & Chu, X. (2020) · arXiv.org · doi:10.48550/arxiv.2003.06307 · Cited by 54 · see citing works
Variance Reduced Local SGD with Lower Communication ComplexityPreprint
Liang, X., Shen, S., Liu, J., Pan, Z., Chen, E., & Cheng, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12844 · Cited by 93 · see citing works
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Wang, J., Tantia, V., Ballas, N., & Rabbat, M. (2019) · arXiv · doi:10.48550/arxiv.1910.00643 · Cited by 90 · see citing works
Federated Learning Over Noisy Channels: Convergence Analysis and Design Examples
Wei, X., & Shen, C. (2022) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2022.3140788 · Cited by 90 · see citing works
Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
Xue, Y., & Qin, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3184700 · Cited by 87 · see citing works
Distributed Learning in the Nonconvex World: From batch data to streaming and beyond
Chang, T., Hong, M., Wai, H.-T., Zhang, X., & Lu, S. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2970170 · Cited by 85 · see citing works
Zhou, T., Zhang, J., & Tsang, D. H. K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3325366 · Cited by 84 · see citing works
Decentralized Federated Learning for Electronic Health Records
Lu, S., Zhang, Y., & Wang, Y. (2020) · Conference on Information Sciences and Systems · doi:10.1109/ciss48834.2020.1570617414 · Cited by 81 · see citing works
Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation
Xu, A. P., Li, W., Guo, P., Yang, D., Roth, H. R., Hatamizadeh, A., Zhao, C., Xu, D., Huang, H., & Xu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02020 · Cited by 77 · see citing works
Adaptive Hierarchical Federated Learning Over Wireless Networks
Xu, B., Xia, W., Wen, W., Liu, P., Zhao, H., & Zhu, H. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3135541 · Cited by 74 · see citing works
Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD
Wang, J., Wang, S., Chen, R., & Ji, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20832 · Cited by 73 · see citing works
Efficient Federated Meta-Learning Over Multi-Access Wireless Networks
Yue, S., Ren, J., Xin, J., Zhang, D.-Y., Zhang, Y., & Zhuang, W. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3143259 · Cited by 47 · see citing works
On the Convergence of Communication-Efficient Local SGD for Federated Learning
Gao, H., Xu, A. P., & Huang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16920 · Cited by 59 · see citing works
Scalable and Low-Latency Federated Learning With Cooperative Mobile Edge Networking
Zhang, Z., Gao, Z., Guo, Y., & Gong, Y. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3216837 · Cited by 55 · see citing works
Communication-Efficient Design for Quantized Decentralized Federated Learning
Chen, L., Liu, W., Chen, Y., & Wang, W. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3363887 · Cited by 55 · see citing works
SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization
Singh, N., Data, D., George, J., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9517986 · Cited by 54 · see citing works
A Primal-Dual SGD Algorithm for Distributed Nonconvex Optimization
Yi, X., Zhang, S., Yang, T., Chai, T., & Johansson, K. H. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105554 · Cited by 53 · see citing works
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
Elgabli, A., Park, J., Bedi, A. S., Issaid, C. B., Bennis, M., & Aggarwal, V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2020.3026398 · Cited by 55 · see citing works
Federated Learning Under Importance Sampling
Rizk, E., Vlaski, S., & Sayed, A. H. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3210365 · Cited by 52 · see citing works
Zeroth-order algorithms for stochastic distributed nonconvex optimization
Yi, X., Zhang, S., Yang, T., & Johansson, K. H. (2022) · Automatica · doi:10.1016/j.automatica.2022.110353 · Cited by 49 · see citing works
SoK: Model Inversion Attack Landscape: Taxonomy, Challenges, and Future Roadmap
Dibbo, S. V. (2023) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf57540.2023.00027 · Cited by 52 · see citing works
Fairness-Aware Federated Learning With Unreliable Links in Resource-Constrained Internet of Things
Li, Z., Zhou, Y., Wu, D., Tang, T., & Wang, R. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3156046 · Cited by 47 · see citing works
Over-the-Air Federated Learning and Optimization
Zhu, J., Shi, Y., Zhou, Y., Jiang, C., Chen, W., & Letaief, K. B. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3352280 · Cited by 42 · see citing works
Local Averaging Helps: Hierarchical Federated Learning and Convergence Analysis.Preprint
Wang, J., Wang, S., Chen, R., & Ji, M. (2020) · arXiv (Cornell University) · Cited by 42 · see citing works
Accelerating Fair Federated Learning: Adaptive Federated Adam
Ju, L., Zhang, T., Salman Zubair, T. O. O. R., & Hellander, A. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3423648 · Cited by 41 · see citing works
Mitigating Poor Data Quality Impact with Federated Unlearning for Human-Centric Metaverse
Wang, P., Wei, Z., Qi, H., Wan, S., Xiao, Y., Sun, G., & Zhang, Q. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3345388 · Cited by 40 · see citing works
Distributed Stochastic Consensus Optimization With Momentum for Nonconvex Nonsmooth Problems
Wang, Z., Zhang, J., Chang, T., Li, J., & Luo, Z.-Q. T. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3097211 · Cited by 37 · see citing works
Optimal Decentralized Distributed Algorithms for Stochastic Convex OptimizationPreprint
Gorbunov, E., Dvinskikh, D., & Gasnikov, A. V. (2019) · arXiv (Cornell University) · doi:10.20347/wias.preprint.2691 · Cited by 38 · see citing works
Asynchronous Decentralized Federated Learning for Heterogeneous Devices
Liao, Y., Xu, Y., Xu, H., Chen, M., Wang, L., & Qiao, C. (2024) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2024.3424444 · Cited by 36 · see citing works
Hyperparameter Optimization for Tomato Leaf Disease Recognition Based on YOLOv11m
Lee, Y.-S., Patil, M. P., Kim, J. G., Seo, Y. B., Ahn, D.-H., & Kim, G. (2025) · Plants · doi:10.3390/plants14050653 · Cited by 39 · see citing works
Ayachi, R., Said, Y., Afif, M., Alshammari, A., Hleili, M., & Abdelali, A. B. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.03.077 · Cited by 34 · see citing works
Adaptive Federated Learning With Negative Inner Product Aggregation
Deng, W., Chen, X., Li, X., & Zhao, H. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3312059 · Cited by 33 · see citing works
Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
Jiang, Z., Wang, W., Li, B., & Yang, Q. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3177222 · Cited by 34 · see citing works
AdaFed: Optimizing Participation-Aware Federated Learning With Adaptive Aggregation Weights
Tan, L., Zhang, X., Zhou, Y., Che, X., Hu, M., Chen, X., & Wu, D. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3168969 · Cited by 32 · see citing works
SloMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Wang, J., Tantia, V., Ballas, N., & Rabbat, M. (2020) · International Conference on Learning Representations · Cited by 31 · see citing works
Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms
Lu, K., Wang, H., Zhang, H., & Wang, L. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2023.3327319 · Cited by 31 · see citing works
Live Gradient Compensation for Evading Stragglers in Distributed Learning
Xu, J., Huang, S., Song, L., & Lan, T. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488815 · Cited by 31 · see citing works
Shen, X., Chen, J., Zhu, S., & Yan, R. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122302 · Cited by 30 · see citing works
Rizk, E., Vlaski, S., & Sayed, A. H. (2020) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc48557.2020.9154327 · Cited by 33 · see citing works
Byzantine-Resilient SGD in High Dimensions on Heterogeneous Data
Data, D., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518248 · Cited by 30 · see citing works
Communication-Efficient Local Decentralized SGD MethodsPreprint
Li, X., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09126 · Cited by 30 · see citing works
Federated Learning's Blessing: FedAvg has Linear SpeedupPreprint
Qu, Z., Lin, K., Li, Z., Zhou, J., & Zhou, Z. (2021) · arXiv (Cornell University) · Cited by 30 · see citing works
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Yuan, K., Chen, Y., Huang, X., Zhang, Y., Pan, P., Xu, Y., & Yin, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00302 · Cited by 29 · see citing works
Federated Learning under Arbitrary Communication Patterns
Avdiukhin, D., & Kasiviswanathan, S. P. (2021) · International Conference on Machine Learning · Cited by 29 · see citing works
Efficient Federated Learning Via Local Adaptive Amended Optimizer With Linear Speedup
Sun, Y., Shen, L., Sun, H., Ding, L., & Tao, D. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3300886 · Cited by 28 · see citing works
Adaptive Heterogeneous Client Sampling for Federated Learning Over Wireless Networks
Luo, B., Xiao, W., Wang, S., Huang, J., & Tassiulas, L. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3368473 · Cited by 28 · see citing works
InSS: An Intelligent Scheduling Orchestrator for Multi-GPU Inference With Spatio-Temporal Sharing
Han, Z., Zhou, R., Xu, C., Zeng, Y., & Zhang, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3430063 · Cited by 28 · see citing works
Distributed Stochastic Gradient Descent with Event-Triggered Communication
George, J., & Gurram, P. K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6206 · Cited by 28 · see citing works
ESync: Accelerating Intra-Domain Federated Learning in Heterogeneous Data Centers
Li, Z., Zhou, H., Zhou, T., Yu, H., Xu, Z., & Sun, G. (2020) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2020.3044043 · Cited by 28 · see citing works
Self-citationSecure Federated Submodel Learning
Niu, C., Wu, F., Tang, S., Hua, L., Jia, R., Lv, C., Wu, Z., & Chen, G. (2019) · arXiv · doi:10.48550/arxiv.1911.02254 · Cited by 28 · see citing works
Cross-Gradient Aggregation for Decentralized Learning from Non-IID dataPreprint
Esfandiari, Y., Tan, S. Y., Jiang, Z., Balu, A., Herron, E., Hegde, C., & Sarkar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02051 · Cited by 28 · see citing works
Recent Theoretical Advances in Decentralized Distributed Convex Optimization
Gorbunov, E., Rogozin, A., Beznosikov, A. N., Dvinskikh, D., & Gasnikov, A. V. (2022) · Springer optimization and its applications · doi:10.1007/978-3-031-00832-0_8 · Cited by 30 · see citing works
Enhancing Decentralized Federated Learning for Non-IID Data on Heterogeneous Devices
Chen, M., Xu, Y., Xu, H., & Huang, L. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00177 · Cited by 27 · see citing works
What Do We Mean by Generalization in Federated Learning?Preprint
Yuan, H., Morningstar, W. R., Ning, L., & Singhal, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14216 · Cited by 27 · see citing works
FedSBS: Federated-Learning participant-selection method for Intrusion Detection Systems
Neto, H. N. C., Hribar, J., Dusparić, I., Fernandes, N. C., & Mattos, D. M. F. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110351 · Cited by 26 · see citing works
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
Elgabli, A., Park, J., Bedi, A. S., Bennis, M., & Aggarwal, V. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054491 · Cited by 26 · see citing works
Coordinating Momenta for Cross-Silo Federated Learning
Xu, A. P., & Huang, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20853 · Cited by 26 · see citing works
Utility Optimization of Federated Learning with Differential Privacy
Zhao, J., Mao, K., Huang, C., & Zeng, Y. (2021) · Discrete Dynamics in Nature and Society · doi:10.1155/2021/3344862 · Cited by 25 · see citing works
Federated Learning with Communication Delay in Edge Networks
Lin, F. P.-C., Brinton, C. G., & Michelusi, N. (2020) · Globecom · doi:10.1109/globecom42002.2020.9322592 · Cited by 24 · see citing works
Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk MinimizationPreprint
Zhang, J., & You, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02712 · Cited by 33 · see citing works
Anarchic Federated LearningPreprint
Yang, H., Zhang, X., Khanduri, P., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09875 · Cited by 24 · see citing works
Toward Communication Efficient Adaptive Gradient Method
Chen, X., Li, X., & Li, P. (2020) · ACM-IMS Foundations of Data Science Conference (FODS) · doi:10.1145/3412815.3416891 · Cited by 23 · see citing works
Sparse Personalized Federated Learning
Liu, X., Li, Y., Wang, Q., Zhang, X., Shao, Y., & Geng, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250658 · Cited by 24 · see citing works
Learn electronic health records by fully decentralized federated learning
Lu, S., Zhang, Y., Wang, Y., & Mack, C. (2019) · arXiv · doi:10.48550/arxiv.1912.01792 · Cited by 23 · see citing works
A Family of Distributed Momentum Methods Over Directed Graphs With Linear Convergence
Gao, J., Liu, X., Dai, Y., Huang, Y., & Yang, P. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3160684 · Cited by 22 · see citing works
Fan, X., Wang, H., Zhao, Y., Li, Y., & Tsui, K. (2021) · Sensors · doi:10.3390/s21051595 · Cited by 21 · see citing works
CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
Yang, H., Liu, J., & Bentley, E. S. (2021) · International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) · doi:10.23919/wiopt52861.2021.9589061 · Cited by 21 · see citing works
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning RatesPreprint
Xie, C., Koyejo, S., Gupta, I., & Lin, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09030 · Cited by 21 · see citing works
Li, H., Li, Z., Li, K., Rellermeyer, J. S., Chen, L. Y., & Li, K. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3047460 · Cited by 20 · see citing works
Krouka, M., Elgabli, A., Issaid, C. B., & Bennis, M. (2022) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2022.3173420 · Cited by 22 · see citing works
GraphCS: Graph-based client selection for heterogeneity in federated learning
Chang, T., Li, L., Wu, M., Yu, W., Wang, X., & Xu, C. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2023.03.003 · Cited by 20 · see citing works
- Authors
- Yu, H., Jin, R., & Yang, S.
- Affiliations
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 7184-7193
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03817
- OpenAlex
- W2944542720
- Cited by
- 244
- References
- 0
- Citations by year
- 2026: 1 · 2025: 14 · 2024: 33 · 2023: 42 · 2022: 41 · 2021: 52 · 2020: 42 · 2019: 18 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Control Multi-Agent Systems
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- linear speedup, momentum SGD, communication efficiency, distributed stochastic gradient descent, convergence analysis, deep neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
