Sparse Communication for Distributed Gradient Descent
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates.Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices.This method can be combined with quantization to further improve the compression.We explore different configurations and apply them to neural machine translation and MNIST image classification tasks.Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task.Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sparsification, on the other hand, proposes to only communicate the informative elements of the gradient or model vectors among nodes [128], [129].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tolu Alabi, Jeffrey D Blanchard, Bradley Gordon, and Russel Steinbach. 2012. Fast k-selection algorithms for graphics processing units. Journal of Experi- mental Algorithmics (JEA) 17:4-2.
- Dan Alistarh, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2016. QSGD: randomized quan- tization for communication-optimal stochas- tic gradient descent. CoRR abs/1610.02132. http://arxiv.org/abs/1610.02132. · arXiv
- Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project adam: Building an efficient and scalable deep learning training system. In OSDI. volume 14, pages 571- 582.
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems. pages 1223- 1231.
- Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016. Communication quantization for data-parallel training of deep neural networks. In Proceedings of the Workshop on Machine Learn- ing in High Performance Computing Environments. IEEE Press, pages 1-8.
- Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In Program of the 13th International Workshop on Spoken Language Translation (IWSLT 2016).
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
- J. Lei Ba, J. R. Kiros, and G. E. Hinton. 2016. Layer Normalization. ArXiv e-prints .
- Ramón P Ñeco and Mikel L Forcada. 1996. Beyond mealy machines: Learning translators with recurrent neural networks. In Proceedings of the 1996 Inter- national Neural Network Society Annual Meeting. San Diego, California, USA.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings 40th Annual Meeting of the Association for Com- putational Linguistics. Philadelphia, PA, pages 311- 318.
- Rajat Raina, Anand Madhavan, and Andrew Y Ng. 2009. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th an- nual international conference on machine learning. ACM, pages 873-880.
- Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient de- scent. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Sys- tems 24, Curran Associates, Inc., pages 693-701. http://papers.nips.cc/paper/4390-hogwild-a-lock- free-approach-to-parallelizing-stochastic-gradient- descent.pdf.
- Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1-bit stochastic gradi- ent descent and application to data-parallel distributed training of speech DNNs. In Interspeech. https://www.microsoft.com/en- us/research/publication/1-bit-stochastic-gradient- descent-and-application-to-data-parallel- distributed-training-of-speech-dnns/.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation sys- tems for WMT 16. In Proceedings of the ACL 2016 First Conference on Machine Translation (WMT16).
- Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. 2010. Parallelized stochastic gradient descent. In Advances in neural information process- ing systems. pages 2595-2603.
- Daniel Zwillinger and Stephen Kokoska. 1999. CRC standard probability and statistics tables and formu- lae. CRC Press.
Cited by
Items 301–400 of 783
Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach
Wang, B., Fang, J., Li, H., & Eldar, Y. C. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3368751 · Cited by 12
Learned Parameter Compression for Efficient and Privacy-Preserving Federated Learning
Chen, Y., Abrahamyan, L., Hichem Sahli, B., & Deligiannis, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3409191 · Cited by 14
QSFL: Two-Level Communication-Efficient Federated Learning on Mobile Edge Devices
Yi, L., Wang, G., Wang, X., & Liu, X. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3455098 · Cited by 11
Jiang, W. P., Xiao, Z., Qian, L., Qin, S., Feng, G., & Wu, Y. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3568875 · Cited by 12
Pais, V., Rao, S., & Muniyal, B. (2025) · IEEE Access · doi:10.1109/access.2025.3573933 · Cited by 12
Adaptive Model Pruning in Decentralized Federated Learning
Mu, Y., Liu, X., & Ratnarajah, T. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3583472 · Cited by 11
Mohammadi, S., Sinaei, S., Balador, A., & Flammini, F. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7032-2_1 · Cited by 16
HyperFeel: An Efficient Federated Learning Framework Using Hyperdimensional Computing
Li, H., Liu, F., Chen, Y., & Jiang, L. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473907 · Cited by 13
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications
Wilkins, G., Di, S., Calhoun, J. C., Li, Z., Kim, K., Underwood, R., Mortier, R., & Cappello, F. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00060 · Cited by 13
Topology-aware Sparse Allreduce for Large-scale Deep Learning
Nguyen, T. T., Wahib, M., & Takano, R. (2019) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/ipccc47392.2019.8958738 · Cited by 12
Privacy preserving distributed training of neural networks
Nikolaidis, S., & Refanidis, I. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04880-0 · Cited by 10
Communication Efficient SGD via Gradient Sampling with Bayes Prior
Song, L., Zhao, K., Pan, P., Liu, Y., Zhang, Y., Xu, Y., & Jin, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01189 · Cited by 14
A Flexible Research-Oriented Framework for Distributed Training of Deep Neural Networks
Barrachina, S., Castelló, A., Catalán, M., Dolz, M. F., & Mestre, J. I. (2021) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw52791.2021.00110 · Cited by 13
A Novel Adaptive Gradient Compression Approach for Communication-Efficient Federated Learning
Yang, W., Yang, Y., Dang, X., Jiang, H., Zhang, Y., & Xiang, W. (2021) · China Automation Congress (CAC) · doi:10.1109/cac53003.2021.9728013 · Cited by 11
Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning
Becking, D., Kirchhoffer, H., Tech, G., Haase, P., Müller, K., Schwarz, H., & Samek, W. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00380 · Cited by 12
Na, S. H., Hong, H. G., Kim, J., & Shin, S. (2022) · Annual Computer Security Applications Conference · doi:10.1145/3564625.3564657 · Cited by 14
SNAP: A Communication Efficient Distributed Machine Learning Framework for Edge Computing
Zhao, Y., Fan, J., Su, L., Song, T., Wang, S., & Qiao, C. (2020) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs47774.2020.00072 · Cited by 13
FLSGD: free local SGD with parallel synchronization
Ye, Q., Zhou, Y., Shi, M., & Lv, J. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-021-04267-5 · Cited by 10
Optimal MIMO Combining for Blind Federated Edge Learning with Gradient Sparsification
Becirovic, E., Chen, Z., & Larsson, E. G. (2022) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc51304.2022.9834030 · Cited by 12
Exploring Fast and Communication-Efficient Algorithms in Large-scale Distributed NetworksPreprint
Yu, Y., Wu, J., & Huang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08924 · Cited by 23
Federated Random Reshuffling with Compression and Variance ReductionPreprint
Malinovsky, G., & Richtárik, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.03914 · Cited by 11
Top-k Shuffled Differential Privacy Federated Learning for Heterogeneous Data
Xiao, D., Fan, X., & Chen, L. (2025) · Sensors · doi:10.3390/s25051441 · Cited by 9
GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection
Hu, P., Melkonian, J., Tang, W., Zhao, H., & Ma, J. (2025) · neural information processing systems · doi:10.52202/085713-1464 · Cited by 17
RTGA: Robust ternary gradients aggregation for federated learning
Yang, C., Xiao, D., Cao, B., & Wu, W. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.113 · Cited by 9
Next Generation Federated Learning for Edge Devices: An Overview
Zhang, J., Du, Z., Sun, J., Li, A., Tang, M., Wu, Y., Gao, Z., Kuo, M., Li, H. H., & Chen, Y. (2022) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic56439.2022.00012 · Cited by 12
Wang, G., Qi, Q., Han, R., Bai, L., & Choi, J. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3445957 · Cited by 10
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
Han, W., Vargaftik, S., Mitzenmacher, M., Karp, B. N., & Basat, R. B. (2024) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3696348.3696857 · Cited by 15
Compressed Private Aggregation for Scalable and Robust Federated Learning Over Massive Networks
Lang, N., Shlezinger, N., D’Oliveira, R. G. L., & Rouayheb, S. E. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564390 · Cited by 12
Monte Carlo Gradient Quantization
Mordido, G., Keirsbilck, M. V., & Keller, A. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00367 · Cited by 13
A Unified Federated DNNs Framework for Heterogeneous Mobile Devices
Li, X., Li, Y., Li, S., Zhou, Y., Chen, C., & Zheng, Z. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3088867 · Cited by 10
CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
Yu, E., Dong, D., Xu, Y., Ouyang, S., & Liao, X. (2021) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3472456.3472508 · Cited by 9
Improving scalability of parallel CNN training by adaptively adjusting parameter update frequency
Lee, S., Kang, Q., Al-Bahrani, R., Agrawal, A., Choudhary, A., & Liao, W. (2021) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2021.09.005 · Cited by 9
Distributed hierarchical deep optimization for federated learning in mobile edge computing
Zheng, X., Shah, S. B. H., Bashir, A. K., Nawaz, R., & RANA, O. F. (2022) · Computer Communications · doi:10.1016/j.comcom.2022.07.028 · Cited by 9
Chen, C., Li, M., & Yang, C. (2023) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00015 · Cited by 10
An Efficient Task-based All-Reduce for Machine Learning Applications
Li, Z., Davis, J., & Jarvis, S. A. (2017) · doi:10.1145/3146347.3146350 · Cited by 9
BiFeat: Supercharge GNN Training via Graph Feature QuantizationPreprint
Ma, Y., Gong, P., Yi, J., Yao, Z., Li, C., He, Y., & Yan, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.14696 · Cited by 11
A Comprehensive Survey on Joint Resource Allocation Strategies in Federated Edge Learning
Zhang, J., Wu, Q., Fan, P., & Fan, Q. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.057006 · Cited by 10
FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning
Medjadji, C., Alawadi, S., Awaysheh, F. M., Leduc, G., Sylvain Kubler, A., & Traon, Y. L. (2025) · International Conference on Federated Learning Technologies and Applications (FLTA) · doi:10.1109/flta67013.2025.11336403 · Cited by 14
Zang, L., Qin, Y., & Li, R. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945217 · Cited by 11
Joint Online Optimization of Model Training and Analog Aggregation for Wireless Edge Learning
Wang, J., Liang, B., Dong, M., Boudreau, G., & Abou-Zeid, H. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2023.3318474 · Cited by 12
Masked Random Noise for Communication-Efficient Federated Learning
Li, S., Cheng, Y., Wang, H., Tang, X. J., Xu, S., Luo, W., Li, Y., Liu, D., He, X., & Li, R. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680608 · Cited by 14
Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing
Lu, R., Jiang, Y., Mao, Y., Tang, C., Chen, B., Cui, L., & Wang, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3504284 · Cited by 13
Two-Tier Data Packing in RLWE-based Homomorphic Encryption for Secure Federated Learning
Zhou, Y., Zheng, P., Cao, X., & Huang, J. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690191 · Cited by 12
A Novel User Behavior Modeling Scheme for Edge Devices with Dynamic Privacy Budget Allocation
Zhang, H., Huang, H., & Peng, C. (2025) · Electronics · doi:10.3390/electronics14050954 · Cited by 8
ZenFlow: Enabling Stall-Free Offloading for LLM Training
Lan, T., Wu, Y., Ma, B., Su, Z., Yang, R., Biçer, T., Tanaka, M., Ruwase, O., Li, D., & Cheng, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3837130 · Cited by 11
Communication Scheduling Optimization for Distributed Deep Learning Systems
Tsai, C.-Y., Lin, C., Liu, P., & Wu, J. (2018) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/padsw.2018.8644593 · Cited by 8
Entropy-Based Gradient Compression for Distributed Deep Learning
Kuang, D., Chen, M., Xiao, D., & Wu, W. (2019) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc/smartcity/dss.2019.00046 · Cited by 8
Akter, S. N., & Adnan, M. A. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403097 · Cited by 8
Communication cost reduction using sparse ternary compression and encoding for FedAvg
Dinh, T. Q. K., Tran, T.-H., & Le, T. (2021) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc52510.2021.9620887 · Cited by 9
Communication-Efficient Semihierarchical Federated Analytics in IoT Networks
Zhao, L., Valero, M., Pouriyeh, S., Li, L., & Sheng, Q. Z. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3137757 · Cited by 9
CP-SGD: Distributed stochastic gradient descent with compression and periodic compensation
Yu, E., Dong, D., Xu, Y., Ouyang, S., & Liao, X. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.05.014 · Cited by 8
Enabling In-Network Floating-Point Arithmetic for Efficient Computation Offloading
Cui, P., Pan, H., Li, Z., Zhang, P., Miao, T., Zhou, J., Guan, H., & Xie, G. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3208425 · Cited by 8
CPA: Compressed Private Aggregation for Scalable Federated Learning Over Massive Networks
Lang, N., Sofer, E., Shlezinger, N., D’Oliveira, R. G. L., & Rouayheb, S. E. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096363 · Cited by 10
Dual-Way Gradient Sparsification for Asynchronous Distributed Deep Learning
Yan, Z., Xiao, D., Chen, M., Zhou, J., & Wu, W. (2020) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3404397.3404401 · Cited by 9
Trends and Advancements in Deep Neural Network CommunicationPreprint
Sattler, F., Wiegand, T., & Samek, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03320 · Cited by 9
DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep LearningPreprint
Kostopoulou, K., Xu, H., Dutta, A., Li, X., Ntoulas, A., & Kalnis, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03112 · Cited by 9
Xiang, Y., Wu, Z., Gong, W., Ding, S., Mo, X., Liu, Y., Wang, S., Liu, P., Hou, Y., Li, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09470 · Cited by 8
Communication-Efficient Distributionally Robust Decentralized Learning
Zecchin, M., Kountouris, M., & Gesbert, D. (2022) · Transactions on machine learning research · doi:10.48550/arxiv.2205.15614 · Cited by 9
Lee, S., & Jang, H. (2025) · IEEE Access · doi:10.1109/access.2025.3547627 · Cited by 10
Edge-Aware Federated AI: Scalable LLM Integration for Privacy-Preserving Big Data Networks
Jonnalagadda, A. K., Natarajan, G. N., Veerapaneni, S. M., & Vikram, S. (2025) · International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) · doi:10.1109/iceccme64568.2025.11277672 · Cited by 8
S 3 GD-MV: Sparse-SignSGD with Majority Vote for Communication-Efficient Distributed Learning
Park, C., & Lee, N. (2023) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit54713.2023.10206480 · Cited by 10
NAIR: An Efficient Distributed Deep Learning Architecture for Resource Constrained IoT System
Xiao, Y., Zhang, D., Wang, Y., Dai, X., Huang, Z., Zhang, W., Yang, Y., Anjum, A., & Qin, F. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3375924 · Cited by 8
SparDL: Distributed Deep Learning Training with Efficient Sparse Communication
Zhao, M., Yin, Y., Mao, Y., Liu, Q., Lü, C., & Gao, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00142 · Cited by 8
Wu, D., Yang, W., Zou, X., Tao, D., Li, S., Xia, W., & Fang, B. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3447221 · Cited by 8
ADTopk: All-Dimension Top-k Compression for High-Performance Data-Parallel DNN Training
Ming, Z., Hu, Y., Zhou, W., Zheng, X., Yao, C., & Feng, D. (2024) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3625549.3658678 · Cited by 10
Secure Collaborative Learning for Self-Adaptive Systems on Connected Autonomous Vehicles
Wu, X., Liu, Y., Chi, X., Jiang, R., Zhou, X., Rafique, W., & Khan, M. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3690768 · Cited by 10
Privacy-Enhanced Federated WiFi Sensing for Health Monitoring in Internet of Things
Lian, Z., Zeng, Q., Liu, Z., Wang, H., Ma, C., Meng, W., Su, C., & Sakurai, K. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3476149 · Cited by 7
Distributed Deep Learning With Gradient Compression for Big Remote Sensing Image Interpretation
Xie, W., Ma, J., Lu, T., Li, Y., Lei, J., Fang, L., & Du, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3517535 · Cited by 10
Review of Mathematical Optimization in Federated Learning
Yang, S., Zhao, F., Zhou, Z., Shi, L., Ren, X., & Xu, A. Z. (2025) · CSIAM Transactions on Applied Mathematics · doi:10.4208//csiam-am.so-2024-0023 · Cited by 9
Communication Efficient Distributed Training with Distributed Lion
Liu, B., Wu, L., Chen, L., Liang, K., Zhu, J., Liang, C., Krishnamoorthi, R., & Liu, Q. (2024) · neural information processing systems · doi:10.52202/079017-0583 · Cited by 18
Du, H., Chen, Y., Feng, X., Xiang, Q., & Liu, H. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109517 · Cited by 8
FedOES: An Efficient Federated Learning Approach
Li, Y., Liu, Z., Huang, Y., & Xu, P. (2023) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice58320.2023.10105791 · Cited by 7
Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant
Loroch, D. M., Pfreundt, F., Wehn, N., & Keuper, J. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-14880-5_1 · Cited by 8
Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
Zhang, N., Tao, M., & Wang, J. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518074 · Cited by 7
An Adaptive Distributed Source Coding Design for Distributed Learning
Zhang, N., & Tao, M. (2021) · International Conference on Wireless Communications and Signal Processing (WCSP) · doi:10.1109/wcsp52459.2021.9613333 · Cited by 7
Sparse Communication for Federated Learning
Thonglek, K., Takahashi, K., Ichikawa, K., Nakasan, C., Leelaprute, P., & Iida, H. (2022) · IEEE International Conference on Fog and Edge Computing (ICFEC) · doi:10.1109/icfec54809.2022.00008 · Cited by 11
Qiao, J., Shen, S., Chen, S., Zhang, X., Lan, T., Cheng, X., & Yu, D. (2023) · ACM International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc) · doi:10.1145/3565287.3610268 · Cited by 8
Randomized Reactive Redundancy for Byzantine Fault-Tolerance in Parallelized LearningPreprint
Gupta, N., & Vaidya, N. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09528 · Cited by 7
Low-Rank Training of Deep Neural Networks for Emerging Memory TechnologyPreprint
Gural, A., Nadeau, P., Tikekar, M., & Murmann, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.03887 · Cited by 7
Privacy Amplification Via Bernoulli SamplingPreprint
Imola, J., & Chaudhuri, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10594 · Cited by 9
Deep learning based brain tumour architecture for weight sharing optimization in federated learning
Onaizah, A. N., Xia, Y., Obaid, A. J., & Hussain, K. (2024) · Expert Systems · doi:10.1111/exsy.13643 · Cited by 8
Dynamic Feature-Embedding Communication via Codebook Distillation for Federated Recommendation
Han, M., Liu, J., Li, D., Zhou, J., Li, Y., Gu, N., Zhang, P., Gu, N., & Lu, T. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3841023 · Cited by 6
A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training
Wu, H., Wang, S., Bai, Y., Li, C., Zhou, Q., Yi, J., Yan, F., Chen, R., & Xu, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3266246 · Cited by 7
Shen, M., Meng, J. Y., Xu, K., Yu, S., & Zhu, L. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3403388 · Cited by 10
Liu, Z., Wang, H., & Li, X. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3440959 · Cited by 6
Wang, Y., Wu, R., Li, X., & Kutscher, D. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11133419 · Cited by 6
Modeling and Optimizing the Scaling Performance in Distributed Deep Learning Training
Liu, T., Miao, T., Wu, Q., Li, Z., He, G., Wu, J., Zhang, S., Yang, X., Gareth Tyson, A., & Xie, G. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511981 · Cited by 10
GrCol-PPFL: User-Based Group Collaborative Federated Learning Privacy Protection Framework
Cheng, J., Liu, Z., Shi, Y., Luo, P., & Sheng, V. S. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.032758 · Cited by 6
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification
Yoon, D., & Oh, S. (2023) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3605573.3605609 · Cited by 7
OSP: Boosting Distributed Model Training with 2-stage Synchronization
Chen, Z., Shi, L., Liu, X., Li, J., Liu, S., & Xu, Y. (2023) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3605573.3605650 · Cited by 6
Get More for Less in Decentralized Learning Systems
Dhasade, A., Kermarrec, A.-M., Pires, R. P., Sharma, R. K., Vujasinovic, M., & Wigger, J. (2023) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00067 · Cited by 8
MetaStore: Analyzing Deep Learning Meta-Data at Scale
Zhang, H., Yan, B., Cao, L., Madden, S. R., & Rundensteiner, E. A. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3648160.3648182 · Cited by 10
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications
Wilkins, G., Di, S., Calhoun, J. C., Li, Z., Kim, K., Underwood, R., Mortier, R., & Cappello, F. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00201 · Cited by 7
Qiu, C., & Yang, K. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3487301 · Cited by 7
Model Parameter Prediction Method for Accelerating Distributed DNN Training
Liu, W., Chen, D.-X., Tan, M., Chen, K.-Y., Yin, Y., Shang, W., Li, J., & Cai, J. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110883 · Cited by 7
Communication-efficient federated knowledge graph embedding with entity-wise top-K sparsification
Zhang, X., Zeng, Z., Zhou, X., Niyato, D. T., & Shen, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114147 · Cited by 9
Standard Deviation Based Adaptive Gradient Compression For Distributed Deep Learning
Chen, M., Yan, Z., Ren, J., & Wu, W. (2020) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid49817.2020.00-40 · Cited by 9
Fast and Privacy-Preserving Federated Joint Estimator of Multi-sUGMs
Tan, X. J., Ma, T., & Su, T. (2021) · IEEE Access · doi:10.1109/access.2021.3099400 · Cited by 7
Signal Processing Aspects of Massive MIMO
Becirovic, E. (2022) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789179294540 · Cited by 10
Aging Aware Retraining for Memristor-based Neuromorphic Computing
Ye, W., Zhang, G. L., Li, B., Schlichtmann, U., Zhuo, C., & Yin, X. (2022) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas48785.2022.9937264 · Cited by 6
Metadata
- Authors
- Aji, A. F., & Heafield, K.
- Affiliations
- University of Edinburgh (GB)
- ORCID
- Heafield, Kenneth
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 440-445
- OpenAlex
- W2606891064
- Cited by
- 585
- References
- 0
- Citations by year
- 2026: 57 · 2025: 89 · 2024: 104 · 2023: 88 · 2022: 79 · 2021: 83 · 2020: 54 · 2019: 24 · 2018: 7
- FWCI
- 10.60
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Nvidia
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- sparse communication, distributed gradient descent, stochastic gradient descent, gradient compression, quantization, neural machine translation, convergence rate, communication efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.