The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
Open accessAbstract
Recent developments on large-scale distributed machine learning applications, e.g., deep neural networks, benefit enormously from the advances in distributed non-convex optimization techniques, e.g., distributed Stochastic Gradient Descent (SGD). A series of recent works study the linear speedup property of distributed SGD variants with reduced communication. The linear speedup property enable us to scale out the computing capability by adding more computing nodes into our system. The reduced communication complexity is desirable since communication overhead is often the performance bottleneck in distributed systems. Recently, momentum methods are more and more widely adopted in training machine learning models and can often converge faster and generalize better. For example, many practitioners use distributed SGD with momentum to train deep neural networks with big data. However, it remains unclear whether any distributed momentum SGD possesses the same linear speedup property as distributed SGD and has reduced communication complexity. This paper fills the gap by considering a distributed communication efficient momentum SGD method and proving its linear speedup property.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, most HE schemes require that the secret keys be renewed often (due to e.g. susceptibility to chosen ciphertext attacks [117]).”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [117]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 36
- Aji, A. F. and Heafield, K. Sparse communication for distributed gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2017.
- Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Assran, M., Loizou, N., Ballas, N., and Rabbat, M. Stochastic gradient push for distributed deep learning. arXiv:1811.10792 , 2018. · arXiv
- Chen, K. and Huo, Q. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2016.
- Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(165-202), 2012.
- Ghadimi, E., Feyzmahdavian, H. R., and Johansson, M. Global convergence of the heavy-ball method for convex optimization. arXiv:1412.7457 , 2014. · arXiv
- Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.
- Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. arXiv:1706.02677 , 2017. · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR) , 2016.
- Horn, R. A. and Johnson, C. R. Matrix Analysis . Cambridge University Press, 1985.
- Jiang, P. and Agrawal, G. A linear speedup analysis of distributed deep learning with sparse and quantized communication. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.
- Jiang, Z., Balu, A., Hegde, C., and Sarkar, S. Collaborative deep learning in fixed topology networks. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Li, M., Andersen, D. G., Smola, A. J., and Yu, K. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems (NIPS) , 2014.
- Lian, X., Huang, Y., Li, Y., and Liu, J. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Information Processing Systems (NIPS) , 2015.
- Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Lin, T., Stich, S. U., and Jaggi, M. Don't use large minibatches, use local SGD. arXiv:1808.07217 , 2018. · arXiv
- Mania, H., Pan, X., Papailiopoulos, D., Recht, B., Ramchandran, K., and Jordan, M. I. Perturbed iterate analysis for asynchronous stochastic optimization. SIAM Journal on Optimization , 27(4):2202-2229, 2017.
- McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.
- Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course . Springer Science & Business Media, 2004.
- Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.
- Seide, F. and Agarwal, A. CNTK: Microsoft's open-source deep-learning toolkit. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 2016.
- Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochastic gradient descent and its application to dataparallel distributed training of speech DNNs. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2014.
- Stich, S. U. Local SGD converges fast and communicates little. arXiv:1805.09767 , 2018. · arXiv
- Strom, N. Scalable distributed DNN training using commodity GPU cloud computing. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2015.
- Su, H., Chen, H., and Xu, H. Experiments on parallel training of deep neural network using model averaging. arXiv:1507.01239v3 , 2018. · arXiv
- Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning (ICML) , 2013.
- Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. D 2 : Decentralized training over decentralized data. arXiv:1803.07068 , 2018. · arXiv
- Wang, J. and Joshi, G. Cooperative SGD: A unified framework for the design and analysis of communicationefficient SGD algorithms. arXiv:1808.07576 , 2018a. · arXiv
- Wang, J. and Joshi, G. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD. arXiv:1810.08313 , 2018b. · arXiv
- Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. A unified analysis of stochastic momentum methods for deep learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
- Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. arXiv:1807.06629 , 2018. · arXiv
- Zavriev, S. and Kostyuk, F. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling , 4(4):336-341, 1993.
- Zhou, F. and Cong, G. On the convergence properties of a K -step averaging stochastic gradient descent algorithm for nonconvex optimization. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Self-citationFederated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2007.07481 · Cited by 927
Personalized Federated Learning with Moreau EnvelopesPreprint
Dinh, C. T., Tran, N. H. K., & Nguyen, T. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08848 · Cited by 697
Model Pruning Enables Efficient Federated Learning on Edge Devices
Jiang, Y., Wang, S., Valls, V., Ko, B. J., Lee, W., Leung, K. K., & Tassiulas, L. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3166101 · Cited by 667
Privacy-Preserving Federated Brain Tumour Segmentation
Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M. J., et al. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32692-0_16 · Cited by 578
Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks
Abad, M. S. H., Ozfatura, E., GUndUz, D., & Ercetin, O. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054634 · Cited by 424
Accelerating Federated Learning via Momentum Gradient Descent
Liu, W., Chen, L., Chen, Y., & Zhang, W. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.2975189 · Cited by 374
Fast-Convergent Federated Learning
Nguyen, H. T., Sehwag, V., Hosseinalipour, S., Brinton, C. G., Chiang, M., & Poor, H. V. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036952 · Cited by 290
Qsparse-local-SGD: Distributed SGD with quantization, sparsification, and local computations
Basu, D., Data, D., Karakus, C., & Diggavi, S. N. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2985917 · Cited by 256
FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data
Zhang, X., Hong, M., Dhople, S. V., Yin, W., & Liu, Y. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3115952 · Cited by 235
Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling
Luo, B., Xiao, W., Wang, S., Huang, J., & Tassiulas, L. (2022) · IEEE INFOCOM 2022 - IEEE Conference on Computer Communications · doi:10.1109/infocom48880.2022.9796935 · Cited by 229
Li, L., Shi, D., Hou, R.-H., Li, H., Pan, M., & Han, Z. (2020) · IEEE Conference on Computer Communications · doi:10.1109/infocom42981.2021.9488839 · Cited by 198
Federated Learning with Buffered Asynchronous AggregationPreprint
Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Esmaeili, M. M., & Huba, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06639 · Cited by 182
A Field Guide to Federated OptimizationPreprint
Wang, J., Charles, Z., Xu, Z., Joshi, G., McMahan, H. B., y Arcas, B. A., Al-Shedivat, M., Andrew, G., Avestimehr, S., Daly, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06917 · Cited by 467
Hu, S., Chen, X., Ni, W., Hossain, E., & Wang, X. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3086014 · Cited by 190
On the Convergence of Local Descent Methods in Federated LearningPreprint
Haddadpour, F., & Mahdavi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14425 · Cited by 176
Handling privacy-sensitive medical data with federated learning: Challenges and future directions
Aouedi, O., Sacco, A., Piamrat, K., & Marchetto, G. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3185673 · Cited by 165
Tighter Theory for Local SGD on Identical and Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04746 · Cited by 162
Decentralized Federated Learning: Balancing Communication and Computing Costs
Liu, W., Chen, L., & Zhang, W. (2022) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2022.3151242 · Cited by 170
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated LearningPreprint
Yang, H., Fang, M., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11203 · Cited by 202
An Improved Analysis of Stochastic Gradient Descent with MomentumPreprint
Liu, Y., Gao, Y., & Yin, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07989 · Cited by 137
Decentralized Deep Learning with Arbitrary Communication CompressionPreprint
Koloskova, A., Lin, T., Stich, S. U., & Jaggi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09356 · Cited by 163
Federated Accelerated Stochastic Gradient DescentPreprint
Yuan, H., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08950 · Cited by 106
First Analysis of Local GD on Heterogeneous Data
Khaled, A., Mishchenko, K., & Richtárik, P. (2019) · arXiv · doi:10.48550/arxiv.1909.04715 · Cited by 103
Semisupervised Distributed Learning With Non-IID Data for AIoT Service Platform
Chiu, T.-C., Shih, Y.-Y., Pang, A.-C., Wang, C.-S., Weng, W., & Chou, C.-T. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2995162 · Cited by 98
Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering
Xu, J., Huang, S., Song, L., & Lan, T. (2022) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs54860.2022.00120 · Cited by 113
Faster Adaptive Federated Learning
Wu, X., Huang, F., Hu, Z., & Huang, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26235 · Cited by 107
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2020) · arXiv · doi:10.48550/arxiv.2008.03606 · Cited by 96
Communication-Efficient Distributed Deep Learning: A Comprehensive SurveyPreprint
Tang, Z., Shi, S., Wang, W., Li, B., & Chu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06307 · Cited by 169
Variance Reduced Local SGD with Lower Communication ComplexityPreprint
Liang, X., Shen, S., Liu, J., Pan, Z., Chen, E., & Cheng, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12844 · Cited by 138
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Wang, J., Tantia, V., Ballas, N., & Rabbat, M. (2019) · arXiv · doi:10.48550/arxiv.1910.00643 · Cited by 90
Federated Learning Over Noisy Channels: Convergence Analysis and Design Examples
Wei, X., & Shen, C. (2022) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2022.3140788 · Cited by 90
Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
Xue, Y., & Qin, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3184700 · Cited by 87
Distributed Learning in the Nonconvex World: From batch data to streaming and beyond
Chang, T., Hong, M., Wai, H.-T., Zhang, X., & Lu, S. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2970170 · Cited by 89
Zhou, T., Zhang, J., & Tsang, D. H. K. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3325366 · Cited by 106
Decentralized Federated Learning for Electronic Health Records
Lu, S., Zhang, Y., & Wang, Y. (2020) · Conference on Information Sciences and Systems · doi:10.1109/ciss48834.2020.1570617414 · Cited by 81
Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation
Xu, A. P., Li, W., Guo, P., Yang, D., Roth, H. R., Hatamizadeh, A., Zhao, C., Xu, D., Huang, H., & Xu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02020 · Cited by 77
Adaptive Hierarchical Federated Learning Over Wireless Networks
Xu, B., Xia, W., Wen, W., Liu, P., Zhao, H., & Zhu, H. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3135541 · Cited by 76
Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD
Wang, J., Wang, S., Chen, R., & Ji, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20832 · Cited by 73
Efficient Federated Meta-Learning Over Multi-Access Wireless Networks
Yue, S., Ren, J., Xin, J., Zhang, D., Zhang, Y., & Zhuang, W. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3143259 · Cited by 67
On the Convergence of Communication-Efficient Local SGD for Federated Learning
Gao, H., Xu, A. P., & Huang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16920 · Cited by 59
Scalable and Low-Latency Federated Learning With Cooperative Mobile Edge Networking
Zhang, Z., Gao, Z., Guo, Y., & Gong, Y. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3216837 · Cited by 55
Communication-Efficient Design for Quantized Decentralized Federated Learning
Chen, L., Liu, W., Chen, Y., & Wang, W. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3363887 · Cited by 59
SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization
Singh, N., Data, D., George, J., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9517986 · Cited by 56
A Primal-Dual SGD Algorithm for Distributed Nonconvex Optimization
Yi, X., Zhang, S., Yang, T., Chai, T., & Johansson, K. H. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105554 · Cited by 53
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
Elgabli, A., Park, J., Bedi, A. S., Issaid, C. B., Bennis, M., & Aggarwal, V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2020.3026398 · Cited by 58
Federated Learning Under Importance Sampling
Rizk, E., Vlaski, S., & Sayed, A. H. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3210365 · Cited by 52
Zeroth-order algorithms for stochastic distributed nonconvex optimization
Yi, X., Zhang, S., Yang, T., & Johansson, K. H. (2022) · Automatica · doi:10.1016/j.automatica.2022.110353 · Cited by 49
SoK: Model Inversion Attack Landscape: Taxonomy, Challenges, and Future Roadmap
Dibbo, S. V. (2023) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf57540.2023.00027 · Cited by 58
Fairness-Aware Federated Learning With Unreliable Links in Resource-Constrained Internet of Things
Li, Z., Zhou, Y., Wu, D., Tang, T., & Wang, R. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3156046 · Cited by 47
Over-the-Air Federated Learning and Optimization
Zhu, J., Shi, Y., Zhou, Y., Jiang, C., Chen, W., & Letaief, K. B. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3352280 · Cited by 51
Local Averaging Helps: Hierarchical Federated Learning and Convergence Analysis.Preprint
Wang, J., Wang, S., Chen, R., & Ji, M. (2020) · arXiv (Cornell University) · Cited by 42
Accelerating Fair Federated Learning: Adaptive Federated Adam
Ju, L., Zhang, T., Salman Zubair, T. O. O. R., & Hellander, A. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3423648 · Cited by 46
Mitigating Poor Data Quality Impact with Federated Unlearning for Human-Centric Metaverse
Wang, P., Wei, Z., Qi, H., Wan, S., Xiao, Y., Sun, G., & Zhang, Q. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3345388 · Cited by 40
Distributed Stochastic Consensus Optimization With Momentum for Nonconvex Nonsmooth Problems
Wang, Z., Zhang, J., Chang, T., Li, J., & Luo, Z.-Q. T. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3097211 · Cited by 37
Optimal Decentralized Distributed Algorithms for Stochastic Convex OptimizationPreprint
Gorbunov, E., Dvinskikh, D., & Gasnikov, A. V. (2019) · arXiv (Cornell University) · doi:10.20347/wias.preprint.2691 · Cited by 70
Asynchronous Decentralized Federated Learning for Heterogeneous Devices
Liao, Y., Xu, Y., Xu, H., Chen, M., Wang, L., & Qiao, C. (2024) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2024.3424444 · Cited by 36
Hyperparameter Optimization for Tomato Leaf Disease Recognition Based on YOLOv11m
Lee, Y.-S., Patil, M. P., Kim, J. G., Seo, Y. B., Ahn, D.-H., & Kim, G. (2025) · Plants · doi:10.3390/plants14050653 · Cited by 39
Ayachi, R., Said, Y., Afif, M., Alshammari, A., Hleili, M., & Abdelali, A. B. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.03.077 · Cited by 42
Adaptive Federated Learning With Negative Inner Product Aggregation
Deng, W., Chen, X., Li, X., & Zhao, H. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3312059 · Cited by 33
Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
Jiang, Z., Wang, W., Li, B., & Yang, Q. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3177222 · Cited by 36
AdaFed: Optimizing Participation-Aware Federated Learning With Adaptive Aggregation Weights
Tan, L., Zhang, X., Zhou, Y., Che, X., Hu, M., Chen, X., & Wu, D. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3168969 · Cited by 36
SloMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Wang, J., Tantia, V., Ballas, N., & Rabbat, M. (2020) · International Conference on Learning Representations · Cited by 31
Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms
Lu, K., Wang, H., Zhang, H., & Wang, L. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2023.3327319 · Cited by 31
Live Gradient Compensation for Evading Stragglers in Distributed Learning
Xu, J., Huang, S., Song, L., & Lan, T. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488815 · Cited by 34
Shen, X., Chen, J., Zhu, S., & Yan, R. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122302 · Cited by 32
Rizk, E., Vlaski, S., & Sayed, A. H. (2020) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc48557.2020.9154327 · Cited by 33
Byzantine-Resilient SGD in High Dimensions on Heterogeneous Data
Data, D., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518248 · Cited by 30
Communication-Efficient Local Decentralized SGD MethodsPreprint
Li, X., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09126 · Cited by 30
Federated Learning's Blessing: FedAvg has Linear SpeedupPreprint
Qu, Z., Lin, K., Li, Z., Zhou, J., & Zhou, Z. (2021) · arXiv (Cornell University) · Cited by 30
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Yuan, K., Chen, Y., Huang, X., Zhang, Y., Pan, P., Xu, Y., & Yin, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00302 · Cited by 35
Federated Learning under Arbitrary Communication Patterns
Avdiukhin, D., & Kasiviswanathan, S. P. (2021) · International Conference on Machine Learning · Cited by 29
Efficient Federated Learning Via Local Adaptive Amended Optimizer With Linear Speedup
Sun, Y., Shen, L., Sun, H., Ding, L., & Tao, D. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3300886 · Cited by 28
Adaptive Heterogeneous Client Sampling for Federated Learning Over Wireless Networks
Luo, B., Xiao, W., Wang, S., Huang, J., & Tassiulas, L. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3368473 · Cited by 31
InSS: An Intelligent Scheduling Orchestrator for Multi-GPU Inference With Spatio-Temporal Sharing
Han, Z., Zhou, R., Xu, C., Zeng, Y., & Zhang, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3430063 · Cited by 29
Distributed Stochastic Gradient Descent with Event-Triggered Communication
George, J., & Gurram, P. K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6206 · Cited by 28
ESync: Accelerating Intra-Domain Federated Learning in Heterogeneous Data Centers
Li, Z., Zhou, H., Zhou, T., Yu, H., Xu, Z., & Sun, G. (2020) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2020.3044043 · Cited by 28
Self-citationSecure Federated Submodel Learning
Niu, C., Wu, F., Tang, S., Hua, L., Jia, R., Lv, C., Wu, Z., & Chen, G. (2019) · arXiv · doi:10.48550/arxiv.1911.02254 · Cited by 28
Cross-Gradient Aggregation for Decentralized Learning from Non-IID dataPreprint
Esfandiari, Y., Tan, S. Y., Jiang, Z., Balu, A., Herron, E., Hegde, C., & Sarkar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02051 · Cited by 28
Recent Theoretical Advances in Decentralized Distributed Convex Optimization
Gorbunov, E., Rogozin, A., Beznosikov, A. N., Dvinskikh, D., & Gasnikov, A. V. (2022) · Springer optimization and its applications · doi:10.1007/978-3-031-00832-0_8 · Cited by 36
Enhancing Decentralized Federated Learning for Non-IID Data on Heterogeneous Devices
Chen, M., Xu, Y., Xu, H., & Huang, L. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00177 · Cited by 27
What Do We Mean by Generalization in Federated Learning?Preprint
Yuan, H., Morningstar, W. R., Ning, L., & Singhal, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14216 · Cited by 27
FedSBS: Federated-Learning participant-selection method for Intrusion Detection Systems
Neto, H. N. C., Hribar, J., Dusparić, I., Fernandes, N. C., & Mattos, D. M. F. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110351 · Cited by 30
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
Elgabli, A., Park, J., Bedi, A. S., Bennis, M., & Aggarwal, V. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054491 · Cited by 27
Coordinating Momenta for Cross-Silo Federated Learning
Xu, A. P., & Huang, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20853 · Cited by 26
Utility Optimization of Federated Learning with Differential Privacy
Zhao, J., Mao, K., Huang, C., & Zeng, Y. (2021) · Discrete Dynamics in Nature and Society · doi:10.1155/2021/3344862 · Cited by 26
Federated Learning with Communication Delay in Edge Networks
Lin, F. P.-C., Brinton, C. G., & Michelusi, N. (2020) · Globecom · doi:10.1109/globecom42002.2020.9322592 · Cited by 26
Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk MinimizationPreprint
Zhang, J., & You, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02712 · Cited by 48
Anarchic Federated LearningPreprint
Yang, H., Zhang, X., Khanduri, P., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09875 · Cited by 26
Toward Communication Efficient Adaptive Gradient Method
Chen, X., Li, X., & Li, P. (2020) · ACM-IMS Foundations of Data Science Conference (FODS) · doi:10.1145/3412815.3416891 · Cited by 23
Sparse Personalized Federated Learning
Liu, X., Li, Y., Wang, Q., Zhang, X., Shao, Y., & Geng, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250658 · Cited by 26
Learn electronic health records by fully decentralized federated learning
Lu, S., Zhang, Y., Wang, Y., & Mack, C. (2019) · arXiv · doi:10.48550/arxiv.1912.01792 · Cited by 23
A Family of Distributed Momentum Methods Over Directed Graphs With Linear Convergence
Gao, J., Liu, X., Dai, Y., Huang, Y., & Yang, P. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3160684 · Cited by 23
Fan, X., Wang, H., Zhao, Y., Li, Y., & Tsui, K. (2021) · Sensors · doi:10.3390/s21051595 · Cited by 21
CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
Yang, H., Liu, J., & Bentley, E. S. (2021) · International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) · doi:10.23919/wiopt52861.2021.9589061 · Cited by 21
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning RatesPreprint
Xie, C., Koyejo, S., Gupta, I., & Lin, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09030 · Cited by 21
Li, H., Li, Z., Li, K., Rellermeyer, J. S., Chen, L. Y., & Li, K. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3047460 · Cited by 20
Krouka, M., Elgabli, A., Issaid, C. B., & Bennis, M. (2022) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2022.3173420 · Cited by 22
GraphCS: Graph-based client selection for heterogeneity in federated learning
Chang, T., Li, L., Wu, M., Yu, W., Wang, X., & Xu, C. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2023.03.003 · Cited by 20
Metadata
- Authors
- Yu, H., Jin, R., & Yang, S.
- Affiliations
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 7184-7193
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03817
- OpenAlex
- W2944542720
- Cited by
- 244
- References
- 0
- Citations by year
- 2026: 1 · 2025: 14 · 2024: 33 · 2023: 42 · 2022: 41 · 2021: 52 · 2020: 42 · 2019: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- linear speedup, momentum SGD, communication efficiency, distributed stochastic gradient descent, convergence analysis, deep neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
