Sparse Communication for Distributed Gradient Descent
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates.Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices.This method can be combined with quantization to further improve the compression.We explore different configurations and apply them to neural machine translation and MNIST image classification tasks.Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task.Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sparsification, on the other hand, proposes to only communicate the informative elements of the gradient or model vectors among nodes [128], [129].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tolu Alabi, Jeffrey D Blanchard, Bradley Gordon, and Russel Steinbach. 2012. Fast k-selection algorithms for graphics processing units. Journal of Experi- mental Algorithmics (JEA) 17:4-2.
- Dan Alistarh, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2016. QSGD: randomized quan- tization for communication-optimal stochas- tic gradient descent. CoRR abs/1610.02132. http://arxiv.org/abs/1610.02132. · arXiv
- Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project adam: Building an efficient and scalable deep learning training system. In OSDI. volume 14, pages 571- 582.
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems. pages 1223- 1231.
- Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016. Communication quantization for data-parallel training of deep neural networks. In Proceedings of the Workshop on Machine Learn- ing in High Performance Computing Environments. IEEE Press, pages 1-8.
- Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In Program of the 13th International Workshop on Spoken Language Translation (IWSLT 2016).
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
- J. Lei Ba, J. R. Kiros, and G. E. Hinton. 2016. Layer Normalization. ArXiv e-prints .
- Ramón P Ñeco and Mikel L Forcada. 1996. Beyond mealy machines: Learning translators with recurrent neural networks. In Proceedings of the 1996 Inter- national Neural Network Society Annual Meeting. San Diego, California, USA.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings 40th Annual Meeting of the Association for Com- putational Linguistics. Philadelphia, PA, pages 311- 318.
- Rajat Raina, Anand Madhavan, and Andrew Y Ng. 2009. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th an- nual international conference on machine learning. ACM, pages 873-880.
- Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient de- scent. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Sys- tems 24, Curran Associates, Inc., pages 693-701. http://papers.nips.cc/paper/4390-hogwild-a-lock- free-approach-to-parallelizing-stochastic-gradient- descent.pdf.
- Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1-bit stochastic gradi- ent descent and application to data-parallel distributed training of speech DNNs. In Interspeech. https://www.microsoft.com/en- us/research/publication/1-bit-stochastic-gradient- descent-and-application-to-data-parallel- distributed-training-of-speech-dnns/.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation sys- tems for WMT 16. In Proceedings of the ACL 2016 First Conference on Machine Translation (WMT16).
- Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. 2010. Parallelized stochastic gradient descent. In Advances in neural information process- ing systems. pages 2595-2603.
- Daniel Zwillinger and Stephen Kokoska. 1999. CRC standard probability and statistics tables and formu- lae. CRC Press.
Cited by
Items 201–300 of 783
Distributed Learning With Sparsified Gradient Differences
Chen, Y.-C., Blum, R. S., Takáč, M., & Sadler, B. M. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3162989 · Cited by 27
Multiple Parallel Federated Learning via Over-the-Air Computation
Shi, G., Guo, S., Ye, J., Saeed, N., & Dang, S. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3194821 · Cited by 24
Communication-Efficient Federated Learning Over MIMO Multiple Access Channels
Jeon, Y., Amiri, M. M., & Lee, N. (2022) · IEEE Transactions on Communications · doi:10.1109/tcomm.2022.3198433 · Cited by 23
Park, J., Lee, K., Lee, S., Zhang, M., & Ko, J. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610917 · Cited by 30
Lightweight Privacy-Preserving Cross-Cluster Federated Learning With Heterogeneous Data
Chen, Z., Yu, S., Chen, F., Wang, F., Liu, X., & Deng, R. H. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3435476 · Cited by 29
JointSQ: Joint Sparsification-Quantization for Distributed Learning
Xie, W., Li, H., Ma, J., Li, Y., Lei, J., Donglai, L., & Fang, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00552 · Cited by 26
FFT-based Gradient Sparsification for the Distributed Training of Deep Neural Networks
Wang, L., Wu, W., Zhang, J., Liu, H., Bosilca, G., Herlihy, M. P., & Fonseca, R. (2020) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3369583.3392681 · Cited by 29
HTDFormer: Hyperspectral Target Detection Based on Transformer With Distributed Learning
Li, Y., Qin, H., & Xie, W. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3317033 · Cited by 26
Sparse Gradient Compression for Distributed SGD
Sun, H., Shao, Y., Jiang, J., Cui, B., Lei, K., Xu, Y., & Wang, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18579-4_9 · Cited by 30
Periodic Stochastic Gradient Descent with Momentum for Decentralized TrainingPreprint
Gao, H., & Huang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10435 · Cited by 26
Learning to Invert: Simple Adaptive Attacks for Gradient Inversion in Federated LearningPreprint
Wu, R., Chen, X., Guo, C., & Weinberger, K. Q. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10880 · Cited by 58
Training Neural Networks from Scratch with Parallel Low-Rank AdaptersPreprint
Huh, M., Cheung, B., Bernstein, J., Isola, P., & Agrawal, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16828 · Cited by 22
Sun, H., Gui, Z., Guo, S., Qi, Q., Wang, J., & Liao, J. (2021) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2021.3062398 · Cited by 21
Federated Learning With Lossy Distributed Source Coding: Analysis and Optimization
Yang, H., Ding, T., & Yuan, X. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3277882 · Cited by 22
Zhou, X., & Yang, G. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120167 · Cited by 19
EGC: Entropy-based gradient compression for distributed deep learning
Xiao, D., Mei, Y., Kuang, D., Chen, M., Guo, B., & Wu, W. (2020) · Information Sciences · doi:10.1016/j.ins.2020.05.121 · Cited by 22
Zhang, N., Tao, M., Wang, J., & Xu, F. (2022) · IEEE Transactions on Communications · doi:10.1109/tcomm.2022.3224977 · Cited by 19
Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation
Huang, K., Yin, H., Huang, H., & Gao, W. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.13192 · Cited by 27
Coded Over-the-Air Computation for Model Aggregation in Federated Learning
Zhang, N., Tao, M., Wang, J., & Shao, S. (2022) · IEEE Communications Letters · doi:10.1109/lcomm.2022.3208035 · Cited by 18
Liu, Y., Zhao, S., Xiong, L., Liu, Y., & Chen, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26400 · Cited by 26
FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization
Long, Q., Anagnostopoulos, C. B., Parambath, S. P., & Bi, D. (2023) · Proceedings · doi:10.1109/icdm58522.2023.00146 · Cited by 25
Zhou, Y., Shi, M., Li, Y., Sun, Y., Ye, Q., & Lv, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00464 · Cited by 22
Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates
Lang, N., Cohen, A., & Shlezinger, N. (2024) · IEEE Transactions on Communications · doi:10.1109/tcomm.2024.3486979 · Cited by 25
User Scheduling for Federated Learning Through Over-the-Air Computation
Ma, X., Sun, H., Wang, Q., & Hu, R. Q. (2021) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2021-fall52928.2021.9625544 · Cited by 18
ScionFL: Efficient and Robust Secure Quantized Aggregation
Ben-Itzhak, Y., Möllering, H., Pinkas, B., Schneider, T. H., Suresh, A., Tkachenko, O., Vargaftik, S., Weinert, C., Yalame, H., & Yanai, A. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00031 · Cited by 27
Variance Reduction with Sparse GradientsPreprint
Elibol, M., Lei, L., & Jordan, M. I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09623 · Cited by 18
New Bounds For Distributed Mean Estimation and Variance ReductionPreprint
Davies, P. M., Gurunathan, V., Moshrefi, N., Ashkboos, S., & Alistarh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09268 · Cited by 21
FedDD: Toward Communication-Efficient Federated Learning With Differential Parameter Dropout
Feng, Z., Chen, X., Wu, Q., Wu, W. R., Zhang, X., & Huang, Q. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3311188 · Cited by 20
A Guide Through the Zoo of Biased SGD
Demidovich, Y. A., Malinovsky, G., Sokolov, I., & Richtárik, P. (2023) · neural information processing systems · doi:10.52202/075280-1005 · Cited by 17
Sharma, P., Kafle, S., Khanduri, P., Bulusu, S., Rajawat, K., & Varshney, P. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06036 · Cited by 17
Accordion: Adaptive Gradient Communication via Critical Learning Regime IdentificationPreprint
Agarwal, S., Wang, H., Lee, K., Venkataraman, S., & Papailiopoulos, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.16248 · Cited by 18
Pufferfish: Communication-efficient Models At No Extra CostPreprint
Wang, H., Agarwal, S., & Papailiopoulos, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03936 · Cited by 60
Li, Q., Zhang, B., Liang, Y., Zhang, Y., Wu, W., Sun, Y., Ma, L., & Xie, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.04964 · Cited by 23
Federated Learning: A Survey of Core Challenges, Current Methods, and Opportunities
Baduwal, M., Paudel, P., & Chaudhary, V. (2026) · Computers · doi:10.3390/computers15030155 · Cited by 27
Federated learning with ℓ1 regularization
Shi, Y., Zhang, Y., Zhang, P., Xiao, Y., & Niu, L. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.05.030 · Cited by 20
Communication Optimization Algorithms for Distributed Deep Learning Systems: A Survey
Yu, E., Dong, D., & Liao, X. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3323282 · Cited by 26
Gao, Y., Hu, B., Mashhadi, M. B., Jin, A.-L., Xiao, P., & Wu, C. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3331372 · Cited by 17
Reducing Communication Overhead via CEO in Distributed Training
Abdi, A., & Fekri, F. (2019) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc.2019.8815453 · Cited by 16
A Partition Based Gradient Compression Algorithm for Distributed Training in AIoT
Guo, B., Liu, Y., & Zhang, C. (2021) · Sensors · doi:10.3390/s21061943 · Cited by 19
CBFL: A Communication-Efficient Federated Learning Framework From Data Redundancy Perspective
Li, K., & Xiao, C. (2021) · IEEE Systems Journal · doi:10.1109/jsyst.2021.3119152 · Cited by 18
Revealing and Protecting Labels in Distributed Training
Dang, T., Thakkar, O. D., Ramaswamy, S., Mathews, R., Chin, P. S., & Beaufays, F. (2021) · OpenBU (Boston University) · doi:10.48550/arxiv.2111.00556 · Cited by 28
Running sparse and low-precision neural network: When algorithm meets hardware
Li, B., Wen, W., Mao, J., Li, S., Chen, Y., & Li, H. (2018) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2018.8297378 · Cited by 18
Step-Ahead Error Feedback for Distributed Training with Compressed Gradient
Xu, A. P., Huo, Z., & Huang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17254 · Cited by 19
CF4FL: A Communication Framework for Federated Learning in Transportation Systems
Sangdeh, P. K., Li, C., Pirayesh, H., Zhang, S., Zeng, H., & Hou, Y. T. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3221770 · Cited by 18
CosSGD: Communication-Efficient Federated Learning with a Simple Cosine-Based QuantizationPreprint
He, Y., Wang, H., Zenk, M., & Fritz, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08241 · Cited by 18
Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning
Mitra, A., Pappas, G. J., & Hassani, H. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2301.00944 · Cited by 18
Impact of Synchronization Topology on DML Performance: Both Logical Topology and Physical Topology
Wang, S., Geng, J., & Li, D. (2021) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2021.3117042 · Cited by 16
Li, Y., Du, W., Han, L., Zhang, Z., & Liu, T. (2023) · Sensors · doi:10.3390/s23239305 · Cited by 19
Scalable Decentralized Multi-Agent Federated Reinforcement Learning: Challenges and Advances
Myakala, P. K., & Kamatala, S. (2023) · International Journal of Electrical Electronics and Computers · doi:10.22161/eec.86.2 · Cited by 23
Energy-Efficient Active Federated Learning on Non-IID Data
Shullary, M. H., Abdellatif, A. A., & Massoudn, Y. (2022) · IEEE International Midwest Symposium on Circuits and Systems (MWSCAS) · doi:10.1109/mwscas54063.2022.9859532 · Cited by 16
Energy-Efficient DNN Training Processors on Micro-AI Systems
Han, D., Kang, S., Kim, S., Lee, J., & Yoo, H. (2022) · IEEE Open Journal of the Solid-State Circuits Society · doi:10.1109/ojsscs.2022.3219034 · Cited by 17
Fan, C., Zhang, X., Zhao, Y., Liu, Y., & Yu, S. (2023) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2023.3292525 · Cited by 15
Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol
Chen, Z., Shi, L., Liu, X., Ai, X., Liu, S., & Xu, Y. (2023) · International Workshop on Quality of Service · doi:10.1109/iwqos57198.2023.10188699 · Cited by 17
SaPus: Self-Adaptive Parameter Update Strategy for DNN Training on Multi-GPU Clusters
Zhang, Z., & Wang, C. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3118609 · Cited by 14
Zeng, Z., Liu, C., Tang, Z., Li, K., & Li, K. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3187815 · Cited by 17
Shen, Z., Tang, Q., Zhou, T., Zhang, Y., Jia, Z., Yu, D., Zhang, Z., & Li, B. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3315847 · Cited by 16
Dai, L., Gong, L., An, Z., Xu, Y., & Diao, B. (2023) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2023.104811 · Cited by 18
DGGI: Deep Generative Gradient Inversion with diffusion model
Wu, L., Liu, Z., Pu, B., Wei, K., Cao, H., & Yao, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102620 · Cited by 17
Long Live TIME: Improving Lifetime and Security for NVM-Based Training-in-Memory Systems
Cai, Y., Lin, Y., Xia, L., Chen, X., Han, S., Wang, Y., & Yang, H. (2020) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2020.2977079 · Cited by 14
Differentially Quantized Gradient Descent
Lin, C.-Y., Kostina, V., & Hassibi, B. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518254 · Cited by 16
Joint Privacy Enhancement and Quantization in Federated Learning
Lang, N., & Shlezinger, N. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834551 · Cited by 16
MicroSecAgg: Streamlined Single-Server Secure Aggregation
Guo, Y., Polychroniadou, A., Shi, E., Byrd, D., & Balch, T. R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0077 · Cited by 21
Özfatura, E., Zhao, J., & Gündüz, D. (2021) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc51858.2021.9593130 · Cited by 16
Li, Q., Yu, W., Xia, Y., & Pang, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.07505 · Cited by 24
Balancing privacy and performance in healthcare: A federated learning framework for sensitive data
Tanveer, F., Iradat, F., Iqbal, W., AlSagri, H. S., Alhakbani, H. A., Ahmad, A., & Khan, F. A. (2025) · Digital Health · doi:10.1177/20552076251381769 · Cited by 15
Adaptive Period Control for Communication Efficient and Fast Convergent Federated Learning
Tchaye-Kondi, J., Zhai, Y., Shen, J., Telikani, A., & Zhu, L. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3416312 · Cited by 15
Sketch-Based Adaptive Communication Optimization in Federated Learning
Zhang, P., Xu, L., Mei, L., & Xu, C. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3475578 · Cited by 13
Imani, H. R., Anderson, J., Farid, S., Amirany, A., & El‐Ghazawi, T. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3483554 · Cited by 18
Privacy-preserving federated learning in asynchronous environment using homomorphic encryption
Gupta, M., Kumar, M., & Dhir, R. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104116 · Cited by 16
Liu, J., Yan, J., Xu, H., Wang, L., Wang, Z., Huang, J., & Qiao, C. (2025) · IEEE Transactions on Networking · doi:10.1109/ton.2025.3600015 · Cited by 19
Patil, R., Yadav, R., & Mishra, K. (2025) · IEEE Access · doi:10.1109/access.2025.3627925 · Cited by 16
Compression scenarios for Federated Learning in Smart Manufacturing
Nasri, S. A. E. M., Ullah, I., & Madden, M. G. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2022.12.239 · Cited by 14
Federated Learning from Heterogeneous Data via Controlled Air Aggregation with Bayesian Estimation
Gafni, T., Cohen, K., & Eldar, Y. C. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3351469 · Cited by 15
Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning
Itahara, S., Nishio, T., Morikura, M., & Yamamoto, K. (2020) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2020-fall49728.2020.9348439 · Cited by 16
Partitioning sparse deep neural networks for scalable training and inference
Demirci, G. V., & Ferhatosmanoğlu, H. (2021) · International Conference on Supercomputing · doi:10.1145/3447818.3460372 · Cited by 16
Differentially Quantized Gradient Methods
Lin, C.-Y., Kostina, V., & Hassibi, B. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3171173 · Cited by 21
FLZip: An Efficient and Privacy-Preserving Framework for Cross-Silo Federated Learning
Feng, X., & Du, H. (2021) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics53846.2021.00044 · Cited by 13
Gradient Obfuscation Gives a False Sense of Security in Federated Learning
Yue, K., Jin, R., Wong, C.-W., Baron, D., & Dai, H. (2022) · arXiv · doi:10.48550/arxiv.2206.04055 · Cited by 26
“In Top-k compression [2], the client selects the k largest components of the gradient in absolute values and zeros out other entires.”
BackgroundMethodResultInfluentialRevolutionizing Wireless Networks with Federated Learning: A Comprehensive Review
Mahdimahalleh, S. E. (2024) · European Journal of Electrical Engineering and Computer Science · doi:10.24018/ejece.2024.8.6.671 · Cited by 13
Federated Learning for IoV Adaptive Vehicle Clusters: A Dynamic Gradient Compression Strategy
Xing, L., Zhao, P., Gao, J., Wu, H., Ma, H., & Zhang, X. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3538551 · Cited by 13
Enhanced federated learning for secure medical data collaboration
Appiah, B., Osei, I., Frimpong, B. K., Commey, D., Owusu-Agyemang, K., & Assamah, G. (2025) · Journal of Analytical Science & Technology · doi:10.1186/s40543-025-00484-2 · Cited by 12
VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMs
Gu, Z., Fan, Q., Sun, L., Liu, Y., & Ye, X. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737411 · Cited by 16
Wireless Federated Learning with Limited Communication and Differential Privacy
Sonee, A., Rini, S., & Huang, Y.-C. (2021) · Globecom · doi:10.1109/globecom46510.2021.9685320 · Cited by 14
Online Model Updating with Analog Aggregation in Wireless Edge Learning
Wang, J., Dong, M., Liang, B., Boudreau, G., & Abou-Zeid, H. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796860 · Cited by 13
Dynamic layer-wise sparsification for distributed deep learning
Zhang, H., Wu, T., Ma, Z., Li, F., & Liu, J. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.04.022 · Cited by 13
Fan, L., Zhang, X., Zhao, Y., Sood, K., & Yu, S. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3326331 · Cited by 14
RPN: A Residual Pooling Network for Efficient Federated Learning
Huang, A., Yuanyuan, C., Yang, L., Chen, T., & Yang, Q. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200222 · Cited by 13
Ning, W., Sun, H., Fu, X., Yang, X., Qi, Q., Wang, J., Liao, J., & Han, Z. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118396 · Cited by 12
JSDoop and TensorFlow.js: Volunteer Distributed Web Browser-Based Neural Network Training
Morell, J. Á., Camero, A., & Alba, E. (2019) · IEEE Access · doi:10.1109/access.2019.2950287 · Cited by 14
Mutual Information Regularization for Vertical Federated LearningPreprint
Zou, T., Liu, Y., & Zhang, Y.-Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.01142 · Cited by 12
FedFQ: Federated Learning with Fine-Grained QuantizationPreprint
Li, H., Xie, W., Ye, H., Ma, J., Ma, S., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.08977 · Cited by 15
GI-NAS: Boosting Gradient Inversion Attacks Through Adaptive Neural Architecture Search
Yu, W., Hao, F., Chen, B., Sui, X., Chen, C., Wu, H., Xia, S., & Xu, K. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3589127 · Cited by 24
Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster
Xu, J., Wang, J., Qi, Q., Sun, H., Liao, J., & Di, Y. (2021) · Journal of Grid Computing · doi:10.1007/s10723-021-09550-6 · Cited by 11
Accelerating local SGD for non-IID data using variance reduction
Liang, X., Shen, S., Chen, E., Liu, J., Liu, Q., Cheng, Y., & Pan, Z. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-1018-0 · Cited by 11
Communication-Efficient Coded Computing for Distributed Multi-Task Learning
Hu, H., Wu, Y., Shi, Y., Li, S., Jiang, C., & Zhang, W. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3275194 · Cited by 11
SK-Gradient: Efficient Communication for Distributed Machine Learning with Data Sketch
Gui, J., Song, Y., Wang, Z., He, C., & Huang, Q. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00183 · Cited by 15
Towards Communication-Efficient Model Updating for On-Device Session-Based Recommendation
Xia, X., Yu, J., Xu, G., & Yin, H. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615088 · Cited by 11
Liu, J., Shang, F., Liu, Y., Liu, H., Li, Y., & Gong, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681094 · Cited by 51
Communication-Efficient Secure Aggregation for Federated Learning
Ergun, I., Sami, H. U., & Güler, B. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001481 · Cited by 19
Stanza: Layer Separation for Distributed Training in Deep Learning
Wu, X., Xu, H. L., Li, B., & Xiong, Y. (2020) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2020.2985684 · Cited by 12
Metadata
- Authors
- Aji, A. F., & Heafield, K.
- Affiliations
- University of Edinburgh (GB)
- ORCID
- Heafield, Kenneth
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 440-445
- OpenAlex
- W2606891064
- Cited by
- 585
- References
- 0
- Citations by year
- 2026: 57 · 2025: 89 · 2024: 104 · 2023: 88 · 2022: 79 · 2021: 83 · 2020: 54 · 2019: 24 · 2018: 7
- FWCI
- 10.60
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Nvidia
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- sparse communication, distributed gradient descent, stochastic gradient descent, gradient compression, quantization, neural machine translation, convergence rate, communication efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.