The DOI matches its DataCite record.
Sparsified SGD with Memory
Open accessAbstract
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsification techniques to reduce the amount of data that needs to be communicated, for instance by only sending the most significant entries of the stochastic gradient (top-k sparsification). Whilst such schemes showed very promising performance in practice, they have eluded theoretical analysis so far. In this work we analyze Stochastic Gradient Descent (SGD) with k-sparsification or compression (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD when equipped with error compensation (keeping track of accumulated errors in memory). That is, communication can be reduced by a factor of the dimension of the problem (sometimes even more) whilst still converging at the same rate. We present numerical experiments to illustrate the theoretical findings and the better scalability for distributed applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The analysis shows that this scheme converges at the same rate as vanilla SGD when equipped with error compensation (keeping track of accumulated errors in memory).”
From Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019) · cites this paper as [55]
In the citing paper
“In other words, communication can be reduced by a factor of the dimension of the problem (sometimes even more) while still converging at the same rate.”
From Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019) · cites this paper as [55]
Citations
Cites
Still being checked.
Cited by
Sorted by citations
Federated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
“A concise convergence analysis of sparsified SGD is given in [55], where SGD is analyzed with k-sparsification or compression (e.”
MethodSCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2019) · arXiv · doi:10.48550/arxiv.1910.06378 · Cited by 1,734
Self-citationRobust and Communication-Efficient Federated Learning From Non-i.i.d. Data
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944481 · Cited by 1,725
On the Convergence of FedAvg on Non-IID Data
Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019) · arXiv · doi:10.48550/arxiv.1907.02189 · Cited by 1,575
“Assumptions 3 and 4 have been made by the works (Zhang et al., 2013; Stich, 2018; Stich et al., 2018; Yu et al., 2019).”
Sattler, F., Muller, K.-R., & Samek, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3015958 · Cited by 1,355
Local SGD Converges Fast and Communicates LittlePreprint
Stich, S. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09767 · Cited by 679
Self-citationDistributed Learning in Wireless Networks: Recent Progress and Future Challenges
Chen, M., Gündüz, D., Huang, K., Saad, W., Bennis, M., Feljan, A. V., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118346 · Cited by 577
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
From distributed machine learning to federated learning: a survey
Liu, J., Huang, J., Zhou, Y., Li, X., Ji, S., Xiong, H., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01664-x · Cited by 397
Autonomous Vehicles Enabled by the Integration of IoT, Edge Intelligence, 5G, and Blockchain
Biswas, A., & Wang, H.-C. (2023) · Sensors · doi:10.3390/s23041963 · Cited by 309
Qsparse-local-SGD: Distributed SGD with quantization, sparsification, and local computations
Basu, D., Data, D., Karakus, C., & Diggavi, S. N. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2985917 · Cited by 256
A Unified Theory of Decentralized SGD with Changing Topology and Local UpdatesPreprint
Koloskova, A., Loizou, N., Boreiri, S., Jaggi, M., & Stich, S. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10422 · Cited by 258
Self-citationSparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Sattler, F., Wiedemann, S., Muller, K.-R., & Samek, W. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852172 · Cited by 240
(2020) · International Conference on Learning Representations
FetchSGD: Communication-Efficient Federated Learning with SketchingPreprint
Rothchild, D., Panda, A., Ullah, E., Ivkin, N. P., Stoica, I. G., Braverman, V., Gonzalez, J. E., & Arora, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07682 · Cited by 234
Over-the-Air Federated Learning From Heterogeneous Data
Sery, T., Shlezinger, N., Cohen, K., & Eldar, Y. C. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3090323 · Cited by 229
Li, L., Shi, D., Hou, R.-H., Li, H., Pan, M., & Han, Z. (2020) · IEEE Conference on Computer Communications · doi:10.1109/infocom42981.2021.9488839 · Cited by 198
A Field Guide to Federated OptimizationPreprint
Wang, J., Charles, Z., Xu, Z., Joshi, G., McMahan, H. B., y Arcas, B. A., Al-Shedivat, M., Andrew, G., Avestimehr, S., Daly, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06917 · Cited by 467
Self-citationFederated Learning: A signal processing perspective
Gafni, T., Shlezinger, N., Cohen, K., Eldar, Y. C., & Poor, H. V. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3125282 · Cited by 191
PowerSGD: Practical Low-Rank Gradient Compression for Distributed OptimizationPreprint
Vogels, T., Karimireddy, S. P., & Jaggi, M. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1905.13727 · Cited by 497
Self-citationA Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks
Shi, S., Wang, Q., Zhao, K., Tang, Z., Wang, Y., Huang, X. L., & Chu, X. (2019) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00220 · Cited by 175
Applications of Federated Learning; Taxonomy, Challenges, and Research Trends
Shaheen, M., Farooq, M. S., Umer, T., & Kim, B.-S. (2022) · Electronics · doi:10.3390/electronics11040670 · Cited by 157
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated LearningPreprint
Yang, H., Fang, M., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11203 · Cited by 202
Adaptive Federated Learning on Non-IID Data With Resource Constraint
Zhang, J., Guo, S., Qu, Z., Zeng, D., Zhan, Y., Liu, Q., & Akerkar, R. A. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2021.3099723 · Cited by 150
Federated Learning with Compression: Unified Analysis and Sharp GuaranteesPreprint
Haddadpour, F., Kamani, M. M., Mokhtari, A., & Mahdavi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01154 · Cited by 140
Communication-efficient distributed learning: An overview
Cao, X., Başar, T., Diggavi, S., Eldar, Y. C., Letaief, K. B., Poor, H. V., & Zhang, J. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3242710 · Cited by 139
Federated Learning Over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis
Xing, H., Simeone, O., & Bi, S. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118400 · Cited by 130
Fei, J., Ho, C.-Y., Sahu, A. N., Canini, M., & Sapio, A. (2021) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3452296.3472904 · Cited by 138
Dynamic Model Pruning with Feedback
Lin, T., Stich, S. U., Barba, L., Dmitriev, D., & Jaggi, M. (2020) · arXiv · doi:10.48550/arxiv.2006.07253 · Cited by 126
Self-citationAdaptive Control of Local Updating and Model Compression for Efficient Federated Learning
Xu, Y., Liao, Y., Xu, H., Ma, Z., Wang, L., & Liu, J. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3186936 · Cited by 127
Decentralized Deep Learning with Arbitrary Communication CompressionPreprint
Koloskova, A., Lin, T., Stich, S. U., & Jaggi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09356 · Cited by 163
Self-citationA Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
Katare, D., Perino, D., Nurmi, J., Warnier, M., Janssen, M., & Ding, A. Y. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3302474 · Cited by 135
Joint Model Pruning and Device Selection for Communication-Efficient Federated Edge Learning
Liu, S., Yu, G., Yin, R., Yuan, J., Shen, L., & Liu, C. (2021) · IEEE Transactions on Communications · doi:10.1109/tcomm.2021.3124961 · Cited by 106
Federated Accelerated Stochastic Gradient DescentPreprint
Yuan, H., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08950 · Cited by 106
Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning
Sun, J., Chen, T., Giannakis, G. B., Yang, Q., & Yang, Z. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3033286 · Cited by 102
Accelerating Decentralized Federated Learning in Heterogeneous Edge Computing
Wang, L., Xu, Y., Xu, H., Chen, M., & Huang, L. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3178378 · Cited by 96
Trusted AI in Multiagent Systems: An Overview of Privacy and Security for Distributed Learning
Ma, C., Li, J., Wei, K., Liu, B., Ding, M., Yuan, L., Han, Z., & Vincent Poor, H. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3306773 · Cited by 96
Prakash, P., Ding, J., Chen, R., Qin, X., Shu, M., Cui, Q., Guo, Y., & Pan, M. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3145865 · Cited by 98
Communication-Efficient Distributed Deep Learning: A Comprehensive SurveyPreprint
Tang, Z., Shi, S., Wang, W., Li, B., & Chu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06307 · Cited by 169
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92
Stich, S. U., & Karimireddy, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05350 · Cited by 91
Self-citationHiveMind: Towards Cellular Native Machine Learning Model Splitting
Wang, S., Zhang, X., Uchiyama, H., & Matsuda, H. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118403 · Cited by 90
Communication-efficient distributed SGD with SketchingPreprint
Ivkin, N. P., Rothchild, D., Ullah, E., Braverman, V., Stoica, I. G., & Arora, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.04488 · Cited by 86
Fast Convergence Rates of Distributed Subgradient Methods With Adaptive Quantization
Doan, T. T., Maguluri, S. T., & Romberg, J. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2020.3014095 · Cited by 86
vqSGD: Vector Quantized Stochastic Gradient Descent
Gandikota, V., Kane, D. M., Maity, R. K., & Mazumdar, A. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3161620 · Cited by 89
Towards efficient communications in federated learning: A contemporary survey
Zhao, Z., Mao, Y., Liu, Y., Song, L., Ouyang, Y., Chen, X., & Ding, W. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2022.12.053 · Cited by 94
Communication-Efficient Distributed Blockwise Momentum SGD with Error-FeedbackPreprint
Zheng, S., Huang, Z., & Kwok, J. T.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10936 · Cited by 81
Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning
Lee, S., Zhang, T., & Avestimehr, A. S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26023 · Cited by 84
Federated Learning Via Inexact ADMM
Zhou, S.-L., & Li, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243080 · Cited by 104
Joint Optimization of Communications and Federated Learning Over the Air
Fan, X., Wang, Y., Huo, Y., & Tian, Z. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3130111 · Cited by 74
Stochastic Distributed Learning with Gradient Quantization and Variance ReductionPreprint
Horváth, S., Kovalev, D. A., Mishchenko, K., Stich, S. U., & Richtárik, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05115 · Cited by 75
Self-citationLearned Gradient Compression for Distributed Deep Learning
Abrahamyan, L., Chen, Y., Bekoulis, G., & Deligiannis, N. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084806 · Cited by 79
Acceleration for Compressed Gradient Descent in Distributed and Federated OptimizationPreprint
Li, Z., Kovalev, D. A., Qian, X., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11364 · Cited by 72
GRACE: A Compressed Communication Framework for Distributed Machine Learning
Xu, H., Ho, C.-Y., Abdelmoniem, A. M., Dutta, A., Bergou, E. H., Karatsenidis, K., Canini, M., & Kalnis, P. (2021) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs51616.2021.00060 · Cited by 75
Magnússon, S., Shokri‐Ghadikolaei, H., & Li, N. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3031073 · Cited by 77
Understanding Top-k Sparsification in Distributed Deep LearningPreprint
Shi, S., Chu, X., Cheung, K. C., & See, S. C. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08772 · Cited by 123
Communication optimization strategies for distributed deep neural network training: A survey
Ouyang, S., Dong, D., Xu, Y., & Xiao, L. (2020) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2020.11.005 · Cited by 73
1-Bit Compressive Sensing for Efficient Federated Learning Over the Air
Fan, X., Wang, Y., Huo, Y., & Tian, Z. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3209190 · Cited by 67
FedLab: A Flexible Federated Learning FrameworkPreprint
Zeng, D., Liang, S., Hu, X., Wang, H., & Xu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11621 · Cited by 78
Optimal Rate Adaption in Federated Learning with Compressed Communications
Cui, L., Su, X., Zhou, Y., & Liu, J. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796982 · Cited by 63
Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints
Ribero, M., Vikalo, H., & Veciana, G. A. D. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3224590 · Cited by 60
On the Convergence of Communication-Efficient Local SGD for Federated Learning
Gao, H., Xu, A. P., & Huang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16920 · Cited by 59
A federated learning framework for healthcare IoT devices
Yuan, B., Ge, S., & Xing, W. (2020) · arXiv · doi:10.48550/arxiv.2005.05083 · Cited by 59
Time-Correlated Sparsification for Communication-Efficient Federated Learning
Özfatura, E., Özfatura, K., & Gündüz, D. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518221 · Cited by 61
A Compressed Gradient Tracking Method for Decentralized Optimization With Linear Convergence
Liao, Y., Li, Z., Huang, K., & Pu, S. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3180695 · Cited by 68
Secure Federated Averaging Algorithm with Differential Privacy
Li, Y., Chang, T., & Chi, C. (2020) · Machine learning for signal processing ... · doi:10.1109/mlsp49062.2020.9231531 · Cited by 56
Continuous Training and Deployment of Deep Learning Models
Prapas, I., Derakhshan, B., Mahdiraji, A. R., & Markl, V. (2021) · Datenbank-Spektrum · doi:10.1007/s13222-021-00386-8 · Cited by 59
Communication-Efficient Design for Quantized Decentralized Federated Learning
Chen, L., Liu, W., Chen, Y., & Wang, W. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3363887 · Cited by 59
Unified Optimal Analysis of the (Stochastic) Gradient MethodPreprint
Stich, S. U. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04232 · Cited by 55
Self-citationSQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization
Singh, N., Data, D., George, J., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9517986 · Cited by 56
Partial Synchronization to Accelerate Federated Learning Over Relay-Assisted Edge Networks
Qu, Z., Guo, S., Wang, H., Ye, B., Wang, Y., Zomaya, A. Y., & Tang, B. (2021) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3083154 · Cited by 56
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Optimization
Singh, N., Data, D., George, J., & Diggavi, S. N. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3145576 · Cited by 53
Elastic Aggregation for Federated Optimization
Chen, D., Hu, J., Tan, V. J., Wei, X., & Wu, E. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01173 · Cited by 56
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
Elgabli, A., Park, J., Bedi, A. S., Issaid, C. B., Bennis, M., & Aggarwal, V. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2020.3026398 · Cited by 58
Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
Cui, L., Su, X., Zhou, Y., & Pan, Y. F. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3087262 · Cited by 53
Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse GradientsPreprint
Mitra, A., Jaafar, R. H., Pappas, G. J., & Hassani, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07053 · Cited by 53
Shuffled Model of Federated Learning: Privacy, Accuracy and Communication Trade-Offs
Girgis, A. M., Data, D., Diggavi, S. N., Kairouz, P., & Suresh, A. T. (2021) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2021.3056102 · Cited by 80
Massive Digital Over-the-Air Computation for Communication-Efficient Federated Edge Learning
Qiao, L., Gao, Z., Mashhadi, M. B., & Gündüz, D. (2024) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2024.3431572 · Cited by 73
A Comprehensive Survey on Training Acceleration for Large Machine Learning Models in IoT
Wang, H., Qu, Z., Zhou, Q., Zhang, H., Luo, B., Xu, W., Guo, S., & Li, R. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3111624 · Cited by 54
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!Preprint
Kovalev, D. A., Koloskova, A., Jaggi, M., Richtárik, P., & Stich, S. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01697 · Cited by 47
Self-citationAn Efficient Statistical-based Gradient Compression Technique for Distributed Training SystemsPreprint
Abdelmoniem, A. M., Elzanaty, A., Alouini, M., & Canini, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10761 · Cited by 50
Communication-Efficient Federated Edge Learning via Optimal Probabilistic Device Scheduling
Zhang, M., Zhu, G., Wang, S., Jiang, J., Liao, Q., Zhong, C., & Cui, S. R. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3166941 · Cited by 47
Federated Learning With Client Selection and Gradient Compression in Heterogeneous Edge Systems
Xu, Y., Jiang, Z., Xu, H., Wang, Z., Qian, C., & Qiao, C. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3309497 · Cited by 52
Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression
Jiang, Z., Xu, Y., Xu, H., Wang, Z., & Qian, C. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10229029 · Cited by 47
A Communication-Efficient Federated Learning Scheme for IoT-Based Traffic Forecasting
Zhang, C., Cui, L., Yu, S., & Yu, J. J. Q. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3132363 · Cited by 49
Stochastic-Sign SGD for Federated Learning with Theoretical GuaranteesPreprint
Jin, R., Huang, Y., He, X., Dai, H., & Wu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10940 · Cited by 87
On Model Transmission Strategies in Federated Learning With Lossy Communications
Su, X., Zhou, Y., Cui, L., & Liu, J. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3240883 · Cited by 47
On the Effectiveness of Partial Variance Reduction in Federated Learning with Heterogeneous Data
Li, B., Schmidt, M. N., Alstrøm, T. S., & Stich, S. U. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00386 · Cited by 45
Self-citationrTop-k: A Statistical Estimation Approach to Distributed SGD
Barnes, L. P., Inan, H. A., Isik, B., & Ozgur, A. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.3042094 · Cited by 44
Adaptive Network Pruning for Wireless Federated Learning
Liu, S., Yu, G., Yin, R., & Yuan, J. (2021) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2021.3074605 · Cited by 43
Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach
Zhang, J.-S., Li, N., & Dedeoglu, M. (2021) · IEEE Conference on Computer Communications · doi:10.1109/infocom42981.2021.9488818 · Cited by 47
A Fast Blockchain-Based Federated Learning Framework With Compressed Communications
Cui, L., Su, X., & Zhou, Y. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3213345 · Cited by 47
AC-SGD: Adaptively Compressed SGD for Communication-Efficient Distributed Learning
Yan, G., Li, T. K., Huang, S., Lan, T., & Song, L. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3192050 · Cited by 40
Federated Learning Over Wireless Networks: Challenges and Solutions
Beitollahi, M., & Lu, N. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3285868 · Cited by 45
YOGA: Adaptive Layer-Wise Model Aggregation for Decentralized Federated Learning
Liu, J., Liu, J., Xu, H., Liao, Y., Wang, Z., & Ma, Q. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2023.3329005 · Cited by 40
Gradient Compression and Correlation Driven Federated Learning for Wireless Traffic Prediction
Zhang, C., Zhang, H., Dang, S., Shihada, B., & Alouini, M. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3524183 · Cited by 40
Jin, H., Wu, D., Zhang, S., Zou, X., Jin, S., Tao, D., Liao, Q., & Xia, W. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3230840 · Cited by 42
Hashemi, A., Acharya, A., Das, R., Vikalo, H., Sanghavi, S., & Dhillon, I. S. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3138977 · Cited by 53
Gradient Obfuscation Gives a False Sense of Security in Federated Learning
Yue, K., Jin, R., Wong, C.-W., Baron, D., & Dai, H. (2022) · arXiv · doi:10.48550/arxiv.2206.04055 · Cited by 26
Metadata
- Authors
- Stich, S. U., Cordonnier, J.-B., & Jaggi, M.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- ORCID
- Jaggi, Martin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 4447-4458
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1809.07599
- OpenAlex
- W2890924858
- Cited by
- 508
- References
- 0
- Citations by year
- 2025: 27 · 2024: 98 · 2023: 97 · 2022: 59 · 2021: 123 · 2020: 75 · 2019: 28 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Signal Processing › Sparse and Compressive Sensing Techniques
- Keywords
- top-k sparsification, communication compression, error compensation, distributed optimization, convergence rate, communication efficiency, memory overhead, large-scale machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
