The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
Open accessAbstract
Recent developments on large-scale distributed machine learning applications, e.g., deep neural networks, benefit enormously from the advances in distributed non-convex optimization techniques, e.g., distributed Stochastic Gradient Descent (SGD). A series of recent works study the linear speedup property of distributed SGD variants with reduced communication. The linear speedup property enable us to scale out the computing capability by adding more computing nodes into our system. The reduced communication complexity is desirable since communication overhead is often the performance bottleneck in distributed systems. Recently, momentum methods are more and more widely adopted in training machine learning models and can often converge faster and generalize better. For example, many practitioners use distributed SGD with momentum to train deep neural networks with big data. However, it remains unclear whether any distributed momentum SGD possesses the same linear speedup property as distributed SGD and has reduced communication complexity. This paper fills the gap by considering a distributed communication efficient momentum SGD method and proving its linear speedup property.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, most HE schemes require that the secret keys be renewed often (due to e.g. susceptibility to chosen ciphertext attacks [117]).”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [117]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 36
- Aji, A. F. and Heafield, K. Sparse communication for distributed gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2017.
- Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Assran, M., Loizou, N., Ballas, N., and Rabbat, M. Stochastic gradient push for distributed deep learning. arXiv:1811.10792 , 2018. · arXiv
- Chen, K. and Huo, Q. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2016.
- Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(165-202), 2012.
- Ghadimi, E., Feyzmahdavian, H. R., and Johansson, M. Global convergence of the heavy-ball method for convex optimization. arXiv:1412.7457 , 2014. · arXiv
- Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.
- Goyal, P., Doll´ ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. arXiv:1706.02677 , 2017. · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR) , 2016.
- Horn, R. A. and Johnson, C. R. Matrix Analysis . Cambridge University Press, 1985.
- Jiang, P. and Agrawal, G. A linear speedup analysis of distributed deep learning with sparse and quantized communication. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.
- Jiang, Z., Balu, A., Hegde, C., and Sarkar, S. Collaborative deep learning in fixed topology networks. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) , 2012.
- Li, M., Andersen, D. G., Smola, A. J., and Yu, K. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems (NIPS) , 2014.
- Lian, X., Huang, Y., Li, Y., and Liu, J. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Information Processing Systems (NIPS) , 2015.
- Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Lin, T., Stich, S. U., and Jaggi, M. Don't use large minibatches, use local SGD. arXiv:1808.07217 , 2018. · arXiv
- Mania, H., Pan, X., Papailiopoulos, D., Recht, B., Ramchandran, K., and Jordan, M. I. Perturbed iterate analysis for asynchronous stochastic optimization. SIAM Journal on Optimization , 27(4):2202-2229, 2017.
- McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.
- Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course . Springer Science & Business Media, 2004.
- Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.
- Seide, F. and Agarwal, A. CNTK: Microsoft's open-source deep-learning toolkit. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 2016.
- Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochastic gradient descent and its application to dataparallel distributed training of speech DNNs. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2014.
- Stich, S. U. Local SGD converges fast and communicates little. arXiv:1805.09767 , 2018. · arXiv
- Strom, N. Scalable distributed DNN training using commodity GPU cloud computing. In Annual Conference of the International Speech Communication Association (INTERSPEECH) , 2015.
- Su, H., Chen, H., and Xu, H. Experiments on parallel training of deep neural network using model averaging. arXiv:1507.01239v3 , 2018. · arXiv
- Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning (ICML) , 2013.
- Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. D 2 : Decentralized training over decentralized data. arXiv:1803.07068 , 2018. · arXiv
- Wang, J. and Joshi, G. Cooperative SGD: A unified framework for the design and analysis of communicationefficient SGD algorithms. arXiv:1808.07576 , 2018a. · arXiv
- Wang, J. and Joshi, G. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD. arXiv:1810.08313 , 2018b. · arXiv
- Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems (NIPS) , 2017.
- Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. A unified analysis of stochastic momentum methods for deep learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
- Yu, H., Yang, S., and Zhu, S. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. arXiv:1807.06629 , 2018. · arXiv
- Zavriev, S. and Kostyuk, F. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling , 4(4):336-341, 1993.
- Zhou, F. and Cong, G. On the convergence properties of a K -step averaging stochastic gradient descent algorithm for nonconvex optimization. In International Joint Conference on Artificial Intelligence (IJCAI) , 2018.
Cited by
Sorted by citations
Items 101–200 of 244
Hierarchical Federated Learning With Momentum Acceleration in Multi-Tier Networks
Yang, Z., Fu, S., Bao, W., Yuan, D., & Zomaya, A. Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3294688 · Cited by 22
Decentralized Deep Learning Using Momentum-Accelerated Consensus
Balu, A., Jiang, Z., Tan, S. Y., Hegde, C., Lee, Y. M., & Sarkar, S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414564 · Cited by 20
Gradual Federated Learning With Simulated Annealing
Nguyen, L. T., Kim, J., & Shim, B. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3125137 · Cited by 21
Adversarial training in communication constrained federated learningPreprint
Shah, D. P., Dube, P., Chakraborty, S., & Verma, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01319 · Cited by 20
Chen, H., Chen, X., Peng, L., & Ma, R. (2023) · Electronics · doi:10.3390/electronics12214463 · Cited by 21
EdgeML: Towards network-accelerated federated learning over wireless edge
Pinyoanuntapong, P., Janakaraj, P., Balakrishnan, R., Lee, M., Chen, C., & Wang, P. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109396 · Cited by 19
On the Convergence of Hybrid Server-Clients Collaborative Training
Yang, K., Chen, S., & Shen, C. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3229443 · Cited by 21
Local Adaptivity in Federated Learning: Convergence and ConsistencyPreprint
Wang, J., Xu, Z., Garrett, Z., Charles, Z., Liu, L., & Joshi, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02305 · Cited by 19
Cheng, J., Luo, P., Xiong, N. N., & Wu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3211564 · Cited by 18
Accelerating Hybrid Federated Learning Convergence Under Partial Participation
Bian, J., Wang, L., Yang, K., Shen, C., & Xu, J. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3408631 · Cited by 24
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic OptimizationPreprint
Singh, N., Data, D., George, J., & Diggavi, S. N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14280 · Cited by 18
Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural NetworksPreprint
Guo, Z., Liu, M., Yuan, Z., Shen, L., Liu, W., & Yang, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02426 · Cited by 19
Zhang, H., Li, C., Dai, W., Zheng, Z., Zou, J., & Xiong, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3469188 · Cited by 17
FedQClip: Accelerating Federated Learning via Quantized Clipped SGD
Qu, Z., Jia, N., Ye, B., Hu, S., & Guo, S. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3477972 · Cited by 21
Robust Hierarchical Federated Learning with Anomaly Detection in Cloud-Edge-End Cooperation Networks
Zhou, Y., Wang, R., Mo, X., Li, Z., & Tang, T. (2022) · Electronics · doi:10.3390/electronics12010112 · Cited by 16
Partial model averaging in Federated Learning: Performance guarantees and benefits
Lee, S., Sahu, A. K., He, C., & Avestimehr, S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126647 · Cited by 16
A Reinforcement Learning Approach for Minimizing Job Completion Time in Clustered Federated Learning
Zhou, R., Yu, J., Wang, R., Li, B., Jiang, J., & Wu, L. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228925 · Cited by 18
Non-IID Federated Learning With Sharper Risk Bound
Wei, B., Li, J., Liu, Y., & Wang, W. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3213187 · Cited by 15
CADA: Communication-Adaptive Distributed AdamPreprint
Chen, T., Guo, Z., Sun, Y., & Yin, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15469 · Cited by 15
Sharp Bounds for Federated Averaging (Local SGD) and Continuous PerspectivePreprint
Glasgow, M., Yuan, H., & Ma, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03741 · Cited by 15
Byzantine-Resilient High-Dimensional Federated Learning
Data, D., & Diggavi, S. N. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3284427 · Cited by 14
Heterogeneous Federated Learning Through Multi-Branch Network
Wang, C.-H., Huang, K.-Y., Chen, J.-C., Shuai, H.-H., & Cheng, W.-H. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428189 · Cited by 15
Decentralized Stochastic Gradient Tracking for Empirical Risk Minimization
Zhang, J., & You, K. (2019) · Cited by 15
Extrapolation for Large-batch Training in Deep LearningPreprint
Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05720 · Cited by 14
Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch AccelerationPreprint
Chen, C., Shen, L., Zou, F., & Liu, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05471 · Cited by 19
FedMoS: Taming Client Drift in Federated Learning with Double Momentum and Adaptive Selection
Wang, X., Chen, Y., Li, Y., Liao, X., Jin, H., & Li, B. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228957 · Cited by 14
FastSlowMo: Federated Learning With Combined Worker and Aggregator Momenta
Yang, Z., Fu, S., Bao, W., Yuan, D., & Zomaya, A. Y. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3187678 · Cited by 13
Convergence Analysis of Distributed Gradient Descent Algorithms With One and Two Momentum Terms
Liu, B., Chai, L., & Yi, J. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3218663 · Cited by 12
Yamany, W., Keshk, M., Moustafa, N., & Turnbull, B. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3319039 · Cited by 13
Liu, L., Zhou, P., Sun, G., Chen, X., Wu, T. T., Yu, H., & Guizani, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127009 · Cited by 12
Self-citationA Communication-Efficient Adaptive Algorithm for Federated Learning Under Cumulative Regret
Salgia, S., Gabay, T., Zhao, Q. Q., & Cohen, K. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3356257 · Cited by 12
Achieving Linear Speedup in Asynchronous Federated Learning With Heterogeneous Clients
Wang, X., Li, Z., Jin, S., & Zhang, J. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3461852 · Cited by 12
Communication Efficient Decentralized Learning Over Bipartite Graphs
Issaid, C. B., Elgabli, A., Park, J., Bennis, M., & Debbah, M. A. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3126859 · Cited by 23
SAM: An Efficient Approach With Selective Aggregation of Models in Federated Learning
Shi, Y., Fan, P., Zhu, Z., Peng, C., Fei, W., & Letaief, K. B. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3373822 · Cited by 12
FedSKETCH: Communication-Efficient and Private Federated Learning via SketchingPreprint
Haddadpour, F., Karimi, B., Li, P., & Li, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.04975 · Cited by 32
Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGDPreprint
Yuan, K., Alghunaim, S. A., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08023 · Cited by 12
SignGuard: Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering.Preprint
Xu, J., Huang, S., Song, L., & Lan, T. (2021) · arXiv (Cornell University) · Cited by 12
Modeling Practically Private Wireless Vehicle to Grid System With Federated Reinforcement Learning
Pokhrel, S. R., Hossain, M. B., & Walid, A. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3344460 · Cited by 11
PAGroup: Privacy-aware grouping framework for high-performance federated learning
Chang, T., Li, L., Wu, M., Yu, W., Wang, X., & Xu, C. (2023) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.12.011 · Cited by 11
Dominating Set Model Aggregation for Communication-Efficient Decentralized Deep Learning
Fotouhi, F., Balu, A., Jiang, Z., Esfandiari, Y., Jahani, S., & Sarkar, S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.057 · Cited by 12
Sharma, P., Kafle, S., Khanduri, P., Bulusu, S., Rajawat, K., & Varshney, P. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06036 · Cited by 11
Buffered Asynchronous SGD for Byzantine LearningPreprint
Yang, Y.-R., & Li, W.-J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00937 · Cited by 15
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMMPreprint
Issaid, C. B., Elgabli, A., Park, J., Bennis, M., & Debbah, M. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06459 · Cited by 11
On Stochastic Moving-Average Estimators for Non-Convex Optimization.Preprint
Guo, Z., Xu, Y., Yin, W., Jin, R., & Yang, T. (2021) · arXiv (Cornell University) · Cited by 11
Self-citationShen, Z., Tang, Q., Zhou, T., Zhang, Y., Jia, Z., Yu, D., Zhang, Z., & Li, B. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3315847 · Cited by 11
PI-Fed: Continual Federated Learning With Parameter-Level Importance Aggregation
Yu, L., Ge, L., Wang, G., Yin, J., Chen, Q., Zhou, J., & He, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3440029 · Cited by 11
Federated Learning Based on Model Discrepancy and Variance Reduction
Zhang, H., Li, C., Dai, W., Zou, J., & Xiong, H. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3517658 · Cited by 10
Federated Learning Model Training Method Based on Data Features Perception Aggregation
Yan, Z., Yi, Y., Zhang, J., Zhao, N., Ren, Y., Wan, J., & Jun, Y. L. (2021) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2021-fall52928.2021.9625291 · Cited by 10
Federated Adam-Type Algorithm for Distributed Optimization With Lazy Strategy
Li, Y., Li, W., Zhang, B., & Du, J. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3175997 · Cited by 10
SGD_Tucker: A Novel Stochastic Optimization Strategy for Parallel Sparse Tucker DecompositionPreprint
Li, H., Li, Z., Li, K., Rellermeyer, J. S., Chen, L. Y., & Li, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03550 · Cited by 10
A unified momentum-based paradigm of decentralized SGD for non-convex models and heterogeneous data
Du, H., Cheng, C., & Ni, C. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104130 · Cited by 9
Communication-Efficient Decentralized Local SGD over Undirected Networks
Qin, T., Etesami, S. R., & Uribe, C. A. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9683272 · Cited by 9
Periodic Stochastic Gradient Descent with Momentum for Decentralized TrainingPreprint
Gao, H., & Huang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10435 · Cited by 18
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep LearningPreprint
Ying, B., Yuan, K., Hu, H., Chen, Y., & Yin, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04287 · Cited by 28
A Unified Linear Speedup Analysis of Federated Averaging and Nesterov FedAvg
Qu, Z., Lin, K., Li, Z., Zhou, J., & Zhou, Z. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15180 · Cited by 48
Medicinal Plants Identification Using Federated Deep Learning
Viet, K. L. D., Ha, K. L., Quoc, T. N., & Hoang, V. T. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.02.171 · Cited by 8
STL-SGD: Speeding Up Local SGD with Stagewise Communication Period
Shen, S., Cheng, Y., Liu, J., & Xu, L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17153 · Cited by 8
Federated Learning for Non-IID Data: From Theory to Algorithm
Wei, B., Li, J., Liu, Y., & Wang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89188-6_3 · Cited by 8
Distributed Optimization over Block-Cyclic DataPreprint
Ding, Y., Niu, C., Yan, Y., Zheng, Z., Wu, F., Chen, G., Tang, S., & Jia, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07454 · Cited by 8
Training Federated GANs with Theoretical Guarantees: A Universal Aggregation ApproachPreprint
Zhang, Y., Qu, H. Y., Chang, Q., Liu, H., Metaxas, D., & Chen, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04655 · Cited by 17
HASFL: Heterogeneity-Aware Split Federated Learning Over Edge Computing Systems
Lin, Z., Chen, Z., Chen, X., Ni, W., & Gao, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3673358 · Cited by 7
Joint Power Control and Data Size Selection for Over-the-Air Computation-Aided Federated Learning
An, X., Fan, R., Zuo, S., Hu, H., Jiang, H., & Zhang, N. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3341958 · Cited by 7
Local AdaGrad-type algorithm for stochastic convex-concave optimization
Liao, L., Shen, L., Duan, J., Kolar, M., & Tao, D. (2022) · Machine Learning · doi:10.1007/s10994-022-06239-z · Cited by 12
FedRL: Improving the Performance of Federated Learning with Non-IID Data
Kang, Y., Li, B., & Zeyl, T. J. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001721 · Cited by 7
Robust communication-efficient decentralized learning with heterogeneity
Zhang, X., Wang, Y., Chen, S., Wang, C., Yu, D., & Cheng, X. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102900 · Cited by 7
Nanor, E., Cobbinah, M. B., Yang, Q., Shao, J., & Kobiah, C. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119725 · Cited by 7
Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient ComputationsPreprint
Khanduri, P., Bulusu, S., Sharma, P., & Varshney, P. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04531 · Cited by 7
Distributed Stochastic Optimization Under a General Variance Condition
Huang, K., Li, X., & Pu, S. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3393169 · Cited by 6
Hierarchical Federated Learning with Adaptive Momentum in Multi-Tier Networks
Yang, Z., Fu, S., Bao, W., Yuan, D., & Zhou, B. B. (2023) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00053 · Cited by 7
Accelerating Communication-Efficient Federated Multi-Task Learning With Personalization and Fairness
Xie, R., Li, C., Zhou, X., & Dong, Z. Y. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3411815 · Cited by 7
Kim, H., Lee, K., Lee, C., Hwang, S.-H., & Youn, C. (2021) · IEEE Access · doi:10.1109/access.2021.3074640 · Cited by 7
Gao, H., Xu, H., & Vučetić, S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/482 · Cited by 6
Personalized Fair Split Learning for Resource-Constrained Internet of Things
Chen, H., Chen, X., Peng, L., & Bai, Y. B. (2023) · Sensors · doi:10.3390/s24010088 · Cited by 7
On Distributed Stochastic Gradient Descent for Nonconvex Functions in the Presence of Byzantines
Bulusu, S., Khanduri, P., Sharma, P., & Varshney, P. K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052956 · Cited by 7
Xia, F., & Chatterjee, R. (2022) · Decision Support Systems · doi:10.1016/j.dss.2022.113766 · Cited by 6
On the Stability Analysis of Open Federated Learning Systems
Sun, Y., Fernando, H., Chen, T., & Shahrampour, S. (2023) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc55779.2023.10156023 · Cited by 6
Defazio, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00406 · Cited by 6
On the Convergence of Decentralized Stochastic Gradient Descent With Biased Gradients
Jiang, Y., Kang, H., Liu, J., & Xu, D. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3531356 · Cited by 5
FedADT: An Adaptive Method Based on Derivative Term for Federated Learning
Gao, H., Wu, Q., Zhao, X., Zhu, J., & Zhang, M. (2023) · Sensors · doi:10.3390/s23136034 · Cited by 8
FedCut: A Spectral Analysis Framework for Reliable Detection of Byzantine Colluders
Gu, H., Fan, L., Tang, X., & Yang, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3375287 · Cited by 5
Mobility-Aware Device Sampling for Statistical Heterogeneity in Hierarchical Federated Learning
Zhang, S., Zheng, Z., Li, Q., Wu, F., & Chen, G. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00067 · Cited by 5
Accelerating decentralized federated learning via momentum GD with heterogeneous delays
Li, N., Shan, H., Song, M., Zhou, Y., Zhao, Z., Yang, H. H., & Hou, F. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100310 · Cited by 5
Unified convergence analysis for adaptive optimization with moving average estimator
Guo, Z., Xu, Y., Yin, W., Jin, R., & Yang, T. (2025) · Machine Learning · doi:10.1007/s10994-024-06650-8 · Cited by 4
Self-citationAsymmetrically Decentralized Federated Learning
Li, Q., Zhang, M., Yin, N., Yin, Q., Shen, L., & Cao, X. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3569185 · Cited by 4
Provable Distributed Stochastic Gradient Descent with Delayed Updates
Gao, H., Wu, G., & Rossi, R. A. (2021) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976700.50 · Cited by 4
Danilova, M., & Malinovskiy, G. S. (2022) · Computer Research and Modeling · doi:10.20537/2076-7633-2022-14-2-277-308 · Cited by 4
Boosting Semi-Supervised Federated Learning with Model Personalization and Client-Variance-Reduction
Wang, S., Xu, Y., Yuan, Y., Wang, X., & Quek, T. Q. S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096678 · Cited by 4
An optimal model using single-dimensional CAE-IRNN based SPOA for cyclone track prediction
Selvaraj, A. R., & Tamilpavai, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120437 · Cited by 4
Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency
Yue, S., Hua, X., Chen, L.-L., & Ren, J. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621260 · Cited by 4
Zhan, W., Wu, G., & Gao, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20884 · Cited by 4
Adaptive Serverless LearningPreprint
Gao, H., & Huang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10422 · Cited by 4
Distributed training of CosPlace for large-scale visual place recognition
Zaccone, R., Berton, G., & Masone, C. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1386464 · Cited by 3
DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models
Saadati, N., Pham, M., Saleem, N., Waite, J. R., Balu, A., Jiang, Z., Hegde, C., & Sarkar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02598 · Cited by 3
Order Optimal Bounds for One-Shot Federated Learning Over Non-Convex Loss Functions
Sharifnassab, A., Salehkaleybar, S., & Golestani, S. J. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3344141 · Cited by 3
Quantized FedPD (QFedPD): Beyond Conventional Wisdom – The Energy Benefits of Frequent Communication
Elgabli, A., Issaid, C. B., Badi, M., & Bennis, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3464239 · Cited by 3
Byzantine Resilient Non-Convex SCSG With Distributed Batch Gradient Computations
Bulusu, S., Khanduri, P., Kafle, S., Sharma, P., & Varshney, P. K. (2021) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2021.3129352 · Cited by 3
L-FGADMM: Layer-Wise Federated Group ADMM for Communication Efficient Decentralized Deep Learning
Elgabli, A., Park, J., Ahmed, S., & Bennis, M. (2020) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc45663.2020.9120758 · Cited by 6
On the Convergence of Hybrid Federated Learning with Server-Clients Collaborative Training
Yang, K., & Shen, C. (2022) · Conference on Information Sciences and Systems · doi:10.1109/ciss53076.2022.9751161 · Cited by 3
PaddleBox: Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising
Zhao, W., Jiao, X., Hu, M., Li, X., Zhang, X., & Li, P. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021133 · Cited by 3
Kaya, E. C., Sahin, M. B., & Hashemi, A. (2023) · IEEE Access · doi:10.1109/access.2023.3284891 · Cited by 3
Metadata
- Authors
- Yu, H., Jin, R., & Yang, S.
- Affiliations
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 7184-7193
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03817
- OpenAlex
- W2944542720
- Cited by
- 244
- References
- 0
- Citations by year
- 2026: 1 · 2025: 14 · 2024: 33 · 2023: 42 · 2022: 41 · 2021: 52 · 2020: 42 · 2019: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- linear speedup, momentum SGD, communication efficiency, distributed stochastic gradient descent, convergence analysis, deep neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
