Sparse Communication for Distributed Gradient Descent
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates.Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices.This method can be combined with quantization to further improve the compression.We explore different configurations and apply them to neural machine translation and MNIST image classification tasks.Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task.Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sparsification, on the other hand, proposes to only communicate the informative elements of the gradient or model vectors among nodes [128], [129].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tolu Alabi, Jeffrey D Blanchard, Bradley Gordon, and Russel Steinbach. 2012. Fast k-selection algorithms for graphics processing units. Journal of Experi- mental Algorithmics (JEA) 17:4-2.
- Dan Alistarh, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2016. QSGD: randomized quan- tization for communication-optimal stochas- tic gradient descent. CoRR abs/1610.02132. http://arxiv.org/abs/1610.02132. · arXiv
- Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project adam: Building an efficient and scalable deep learning training system. In OSDI. volume 14, pages 571- 582.
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems. pages 1223- 1231.
- Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016. Communication quantization for data-parallel training of deep neural networks. In Proceedings of the Workshop on Machine Learn- ing in High Performance Computing Environments. IEEE Press, pages 1-8.
- Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In Program of the 13th International Workshop on Spoken Language Translation (IWSLT 2016).
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
- J. Lei Ba, J. R. Kiros, and G. E. Hinton. 2016. Layer Normalization. ArXiv e-prints .
- Ramón P Ñeco and Mikel L Forcada. 1996. Beyond mealy machines: Learning translators with recurrent neural networks. In Proceedings of the 1996 Inter- national Neural Network Society Annual Meeting. San Diego, California, USA.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings 40th Annual Meeting of the Association for Com- putational Linguistics. Philadelphia, PA, pages 311- 318.
- Rajat Raina, Anand Madhavan, and Andrew Y Ng. 2009. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th an- nual international conference on machine learning. ACM, pages 873-880.
- Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient de- scent. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Sys- tems 24, Curran Associates, Inc., pages 693-701. http://papers.nips.cc/paper/4390-hogwild-a-lock- free-approach-to-parallelizing-stochastic-gradient- descent.pdf.
- Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1-bit stochastic gradi- ent descent and application to data-parallel distributed training of speech DNNs. In Interspeech. https://www.microsoft.com/en- us/research/publication/1-bit-stochastic-gradient- descent-and-application-to-data-parallel- distributed-training-of-speech-dnns/.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation sys- tems for WMT 16. In Proceedings of the ACL 2016 First Conference on Machine Translation (WMT16).
- Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. 2010. Parallelized stochastic gradient descent. In Advances in neural information process- ing systems. pages 2595-2603.
- Daniel Zwillinger and Stephen Kokoska. 1999. CRC standard probability and statistics tables and formu- lae. CRC Press.
Cited by
Items 101–200 of 783
Distributed learning of fully connected neural networks using independent subnet training
Yuan, B., Wolfe, C. R., Dun, C., Tang, Y., Kyrillidis, A., & Jermaine, C. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3529337.3529343 · Cited by 69
Li, A., Marković, M., Edwards, P. J., & Leontidis, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122847 · Cited by 61
FedSCR: Structure-based Communication Reduction for Federated Learning
Wu, X., Yao, X., & Wang, C. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3046250 · Cited by 70
Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
Cui, L., Su, X., Zhou, Y., & Pan, Y. F. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3087262 · Cited by 55
Federated Learning With Client Selection and Gradient Compression in Heterogeneous Edge Systems
Xu, Y., Jiang, Z., Xu, H., Wang, Z., Qian, C., & Qiao, C. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3309497 · Cited by 63
Computation Scheduling for Distributed Machine Learning with Straggling Workers
Amiri, M. M., & Gunduz, D. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682911 · Cited by 49
FLASH: Heterogeneity-Aware Federated Learning at Scale
Yang, C., Xu, M., Wang, Q., Chen, Z., Huang, K., Ma, Y., Bian, K., Huang, G., Liu, Y., Jin, X., et al. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3214234 · Cited by 60
Communication-Efficient Federated Edge Learning via Optimal Probabilistic Device Scheduling
Zhang, M., Zhu, G., Wang, S., Jiang, J., Liao, Q., Zhong, C., & Cui, S. R. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3166941 · Cited by 54
FedDQ: Communication-Efficient Federated Learning with Descending Quantization
Qu, L., Song, S., & Tsui, C.-Y. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001205 · Cited by 55
Communication-Efficient Federated Learning for Resource-Constrained Edge Devices
Lan, G., Liu, X.-Y., Zhang, Y., & Wang, X. (2023) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2023.3309773 · Cited by 70
Communication Efficient Federated Learning over Multiple Access Channels
Chang, W.-T., & Tandon, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08737 · Cited by 43
Gradient Compression Supercharged High-Performance Data Parallel DNN Training
Bai, Y., Li, C., Zhou, Q., Yi, J., Gong, P., Yan, F., Chen, R., & Xu, Y. (2021) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3477132.3483553 · Cited by 68
Communication-Efficient Vertical Federated Learning
Khan, A., ten Thij, M., & Wilbik, A. (2022) · Algorithms · doi:10.3390/a15080273 · Cited by 64
Zhang, X., Zhang, B., Yu, W., & Kang, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3244136 · Cited by 63
GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization
Fang, H., Chen, B., Wang, X., Wang, Z., & Xia, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00458 · Cited by 83
Cai, Y., Lin, Y., Xia, L., Chen, X., Han, S., Wang, Y., & Yang, H. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196071 · Cited by 53
A Distributed Hierarchical Deep Computation Model for Federated Learning in Edge Computing
Zheng, H., Gao, M., Chen, Z. D., & Feng, X. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3065719 · Cited by 50
A Fast Blockchain-Based Federated Learning Framework With Compressed Communications
Cui, L., Su, X., & Zhou, Y. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3213345 · Cited by 48
Mu, Y., Garg, N., & Ratnarajah, T. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3196463 · Cited by 49
Federated Learning Over Wireless Networks: Challenges and Solutions
Beitollahi, M., & Lu, N. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3285868 · Cited by 51
rTop-k: A Statistical Estimation Approach to Distributed SGD
Barnes, L. P., Inan, H. A., Isik, B., & Ozgur, A. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.3042094 · Cited by 44
“Definition 1 (Top-r Sparsification [1], [10], [26], [27]): For a parameter 1 ≤ r ≤ d, the operator topr : Rd → Rd is defined for ω ∈ Rd as ( topr(ω) ) i := { (ω)i, if i ∈ {π(1), .”
Federated Neuromorphic Learning of Spiking Neural Networks for Low-Power Edge Intelligence
Skatchkovsky, N., Jang, H., & Simeone, O. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053861 · Cited by 35
The Communication-Aware Clustered Federated Learning Problem
Shlezinger, N., Rini, S., & Eldar, Y. C. (2020) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit44484.2020.9174245 · Cited by 53
Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach
Zhang, J., Li, N., & Dedeoğlu, M. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488818 · Cited by 53
Distributed Intelligence in Wireless Networks
Liu, X., Yu, J., Liu, Y., Gao, Y., Mahmoodi, T., Lambotharan, S., & Tsang, D. H. K. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3265425 · Cited by 48
Local SGD: Unified Theory and New Efficient MethodsPreprint
Gorbunov, E., Hanzely, F., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02828 · Cited by 57
RedSync: Reducing synchronization bandwidth for distributed deep learning training system
Fang, J., Fu, H., Yang, G., & Hsieh, C. (2019) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2019.05.016 · Cited by 47
Accelerating CNN Training by Pruning Activation Gradients
Ye, X., Dai, P., Luo, J., Guo, X., Qi, Y., Yang, J., & Chen, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_20 · Cited by 51
Secure Aggregation in Federated Learning via Multiparty Homomorphic Encryption
Hosseini, E., & Khisti, A. J. (2021) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps52748.2021.9682053 · Cited by 47
Fan, X., Wang, Y., Huo, Y., & Tian, Z. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3164339 · Cited by 43
Reducing Communication Overhead of Federated Learning through Clustering Analysis
Al-Saedi, A. A., Boeva, V., & Casalicchio, E. (2021) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc53001.2021.9631391 · Cited by 45
Chen, L., Xiao, D., Xiao, X., & Zhang, Y. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3486611 · Cited by 42
Privacy-Preserving Incremental ADMM for Decentralized Consensus Optimization
Ye, Y., Chen, H., Xiao, M., Skoglund, M., & Poor, H. V. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3027917 · Cited by 39
Client selection in federated learning based on gradients importance
Marnissi, O., Hammouti, H. E., & Bergou, E. H. (2024) · AIP conference proceedings · doi:10.1063/5.0194687 · Cited by 38
Double Quantization for Communication-Efficient Distributed OptimizationPreprint
Yu, Y., Wu, J., & Huang, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10111 · Cited by 47
Sparsified Secure Aggregation for Privacy-Preserving Federated LearningPreprint
Ergun, I., Sami, H. U., & Güler, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.12872 · Cited by 47
FedVQCS: Federated Learning via Vector Quantized Compressed Sensing
Oh, Y., Jeon, Y., Chen, M., & Saad, W. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3291877 · Cited by 45
Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism
Gupta, V., Choudhary, D., Tang, P. T. P., Wei, X., Wang, X., Huang, Y., Kejariwal, A., Ramchandran, K., & Mahoney, M. W. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467080 · Cited by 48
Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning
Zhang, C., Xia, J., Yang, B., Puyang, H., Wang, W., Chen, R., Akkuş, İ. E., Aditya, P., & Yan, F. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3486998 · Cited by 50
Over-the-Air Federated Learning via Second-Order Optimization
Yang, P., Jiang, Y., Wang, T., Zhou, Y., Shi, Y., & Jones, C. N. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3185156 · Cited by 41
Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
Jiang, Z., Wang, W., Li, B., & Yang, Q. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3177222 · Cited by 46
AceFL: Federated Learning Accelerating in 6G-Enabled Mobile Edge Computing Networks
He, J., Guo, S., Li, M., & Zhu, Y. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3190330 · Cited by 37
On the Future of Cloud Engineering
Bermbach, D., Chandra, A., Krintz, C., Gokhale, A. S., Slominski, A., Thamsen, L., Cavalcante, E., Guo, T., Brandić, I., & Wolski, R. (2021) · IEEE International Conference on Cloud Engineering · doi:10.1109/ic2e52221.2021.00044 · Cited by 36
Distributed Deep Reinforcement Learning: Learn How to Play Atari Games in 21 minutes
Adamski, I., Adamski, R., Grel, T., Jędrych, A., Kaczmarek, K., & Michalewski, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-92040-5_19 · Cited by 40
Abdi, A., & Fekri, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5706 · Cited by 44
DC2: Delay-aware Compression Control for Distributed Machine Learning
Abdelmoniem, A. M., & Canini, M. (2021) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom42981.2021.9488810 · Cited by 48
Fault tolerance in neuromorphic computing systems
Liu, M., Xia, L., Wang, Y., & Chakrabarty, K. (2019) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3287624.3288743 · Cited by 35
Overlapping Communication With Computation in Parameter Server for Scalable DL Training
Wang, S., Pi, A., Zhou, X., Wang, J., & Xu, C. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3062721 · Cited by 37
The Fundamental Price of Secure Aggregation in Differentially Private Federated LearningPreprint
Chen, W., Choquette-Choo, C. A., Kairouz, P., & Suresh, A. T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.03761 · Cited by 77
Li, M., Chen, Y., Wang, Y., & Pan, Y. (2020) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv50220.2020.9305383 · Cited by 43
Graph Federated Learning for CIoT Devices in Smart Home Applications
Rasti-Meymandi, A., Sheikholeslami, S. M., Abouei, J., & Plataniotis, K. N. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3228727 · Cited by 35
Joint Gradient Sparsification and Device Scheduling for Federated Learning
Lin, X., Liu, Y., Chen, F., Ge, X., & Huang, Y. (2023) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2023.3239373 · Cited by 39
Efficient federated learning on resource-constrained edge devices based on model pruning
Wu, T., Song, C., & Zeng, P. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01120-5 · Cited by 41
COTAF: Convergent Over-the-Air Federated Learning
Sery, T., Shlezinger, N., Cohen, K., & Eldar, Y. C. (2020) · Globecom · doi:10.1109/globecom42002.2020.9322580 · Cited by 34
SparSFA: Towards robust and communication-efficient peer-to-peer federated learning
Wang, H., Muñoz-González, L., Hameed, M. Z., Eklund, D., & Raza, S. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103182 · Cited by 38
Communication compression techniques in distributed deep learning: A survey
Wang, Z., Wen, M., Xu, Y., Zhou, Y., Wang, J. H., & Zhang, L. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102927 · Cited by 48
FedSparse: A Communication-Efficient Federated Learning Framework Based on Sparse Updates
Li, J., Zhang, Y., Li, Y., Gong, X., & Wang, W. (2024) · Electronics · doi:10.3390/electronics13245042 · Cited by 33
CB-DSL: Communication-Efficient and Byzantine-Robust Distributed Swarm Learning on Non-i.i.d. Data
Fan, X., Wang, Y., Huo, Y., & Tian, Z. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3312345 · Cited by 32
Information Compression in the AI Era: Recent Advances and Future Challenges
Chen, J., Fang, Y., Khisti, A. J., Özgür, A., & Shlezinger, N. (2025) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2025.3560359 · Cited by 42
Quantization of Distributed Data for Learning
Hanna, O. A., Ezzeldin, Y. H., Fragouli, C., & Diggavi, S. N. (2021) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2021.3105359 · Cited by 35
Adaptive Sparsification and Quantization for Enhanced Energy Efficiency in Federated Learning
Marnissi, O., Hammouti, H. E., & Bergou, E. H. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3425531 · Cited by 33
Communication-Efficient Federated Multitask Learning Over Wireless Networks
Ma, H., Guo, H., & Lau, V. K. N. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3201310 · Cited by 32
Distributed Deep Learning Optimized System over the Cloud and Smart Phone Devices
Jiang, H., Starkman, J., Lee, Y., Chen, H., Qian, X., & Huang, M.-C. (2019) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2941492 · Cited by 32
Privacy-Encoded Federated Learning Against Gradient-Based Data Reconstruction Attacks
Liu, H., Li, B., Gao, C., Xie, P., & Zhao, C. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3309095 · Cited by 27
Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs
Lim, H., Ye, J., Jyothi, S. A., & Han, D. (2024) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3651890.3672228 · Cited by 37
MIPD: An Adaptive Gradient Sparsification Framework for Distributed DNNs Training
Zhang, Z., & Wang, C. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3154387 · Cited by 46
FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices
Wu, D., Yang, W., Jin, H., Zou, X., Xia, W., & Fang, B. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3307459 · Cited by 29
Wang, B., Fang, J., Li, H., & Zeng, B. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3316588 · Cited by 36
Wang, H., Guo, S., & Li, R. (2019) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3337821.3337828 · Cited by 27
An Information-Theoretic Justification for Model PruningPreprint
Isik, B., Weissman, T., & No, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08329 · Cited by 29
Communication-Efficient Federated Learning Over-the-Air With Sparse One-Bit Quantization
Oh, J., Lee, D., Won, D., Noh, W., & Cho, S. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3432758 · Cited by 29
Wang, L., Xu, X., & Pei, J. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3577323 · Cited by 63
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Optimization
Singh, N., Data, D., George, J., & Diggavi, S. N. (2020) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc42340.2020.9303828 · Cited by 32
Luo, P., Yu, F. R., Chen, J., Li, J., & Leung, V. C. M. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3051611 · Cited by 27
ResFed: Communication-Efficient Federated Learning With Deep Compressed Residuals
Song, R., Zhou, L., Lyu, L., Festag, A., & Knoll, A. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3324079 · Cited by 30
Communication-Censored Distributed Stochastic Gradient Descent
Li, W., Wu, Z., Chen, T., Li, L., & Ling, Q. H. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3083655 · Cited by 31
FedDQ: A communication-efficient federated learning approach for Internet of Vehicles
Mo, Z., Gao, Z., Zhao, C., & Lin, Y. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102690 · Cited by 25
Efficient asynchronous federated learning with sparsification and quantization
Jia, J., Liu, J., Zhou, C., Tian, H., Dong, M., & Dou, D. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8002 · Cited by 32
Deng, Z., Sun, R., Xue, M., Wen, S., Camtepe, S. A., Nepal, ., & Xiang, Y. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3532957 · Cited by 25
Large-Scale Training System for 100-Million Classification at Alibaba
Song, L., Pan, P., Zhao, K., Yang, H., Chen, Y., Zhang, Y., Xu, Y., & Jin, R. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403342 · Cited by 33
Communication-efficient Federated Learning Through 1-Bit Compressive Sensing and Analog Aggregation
Fan, X., Wang, Y., Huo, Y., & Tian, Z. (2021) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops50388.2021.9473872 · Cited by 26
Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
Li, S., Mangoubi, O., Xu, L., & Guo, T. (2021) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs51616.2021.00057 · Cited by 31
Digital Twin-Enabled Efficient Federated Learning for Collision Warning in Intelligent Driving
Tang, L., Wen, M., Shan, Z., Li, L., Liu, Q., & Chen, Q. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3330938 · Cited by 28
Cui, X., Zhang, W., Finkler, U., Saon, G., Picheny, M., & Kung, D. S. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2969859 · Cited by 24
A data parallel approach for distributed neural networks to achieve faster convergence
Nagaraju, C., Ramesh, Y., & Mohan, C. K. (2024) · International Conference on Machine Vision · doi:10.1117/12.3023413 · Cited by 24
Peta-Scale Embedded Photonics Architecture for Distributed Deep Learning Applications
Wu, Z., Dai, L. Y., Novick, A., Glick, M. S., Zhu, Z., Rumley, S., Michelogiannakis, G., Shalf, J. M., & Bergman, K. (2023) · Journal of Lightwave Technology · doi:10.1109/jlt.2023.3276588 · Cited by 24
Two-layer accumulated quantized compression for communication-efficient federated learning: TLAQC
Ren, Y., Cao, Y., Ye, C., & Cheng, X. (2023) · Scientific Reports · doi:10.1038/s41598-023-38916-x · Cited by 27
Sun, X., Ren, X., Ma, S., Wei, B., Li, W., Xu, J., Wang, H., & Zhang, Y. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2883613 · Cited by 28
ClusterGrad: Adaptive Gradient Compression by Clustering in Federated Learning
Cui, L., Su, X., Zhou, Y., & Zhang, L. (2020) · Globecom · doi:10.1109/globecom42002.2020.9322527 · Cited by 27
Quantization and Knowledge Distillation for Efficient Federated Learning on Edge Devices
Qu, X., Wang, J., & Xiao, J. Z. (2020) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-smartcity-dss50907.2020.00129 · Cited by 27
Du, W., Li, A., Zhou, P., Xu, Z., Wang, X., Jiang, H., & Wu, D. O. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2999594 · Cited by 24
Communication-Adaptive Stochastic Gradient Methods for Distributed Learning
Chen, T., Sun, Y., & Yin, W. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3099977 · Cited by 22
Wireless Network Optimization for Federated Learning with Model Compression in Hybrid VLC/RF Systems
Huang, W., Yang, Y., Chen, M., Liu, C., Feng, C., & Poor, H. V. (2021) · Entropy · doi:10.3390/e23111413 · Cited by 22
He, J., Guo, S., Qiao, D., & Lin, Y. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3145360 · Cited by 23
Li, Y., He, Z., Gu, X., Xu, H., & Ren, S. (2022) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2022.2079730 · Cited by 25
FedCS: Communication-Efficient Federated Learning with Compressive Sensing
Liu, Y., Chang, S., & Liu, Y. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads56603.2022.00011 · Cited by 25
Communication-Efficient and Privacy-Preserving Aggregation in Federated Learning With Adaptability
Sun, X., Yuan, Z., Kong, X., Xue, L., He, L., & Ying, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3396217 · Cited by 23
Gradient Statistics Aware Power Control for Over-the-Air Federated Learning in Fading Channels
Zhang, N., & Tao, M. (2020) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops49005.2020.9145260 · Cited by 30
Analog Compression and Communication for Federated Learning over Wireless MAC
Abdi, A., Saidutta, Y. M., & Fekri, F. (2020) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc48557.2020.9154309 · Cited by 22
CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
Yang, H., Liu, J., & Bentley, E. S. (2021) · International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) · doi:10.23919/wiopt52861.2021.9589061 · Cited by 28
Metadata
- Authors
- Aji, A. F., & Heafield, K.
- Affiliations
- University of Edinburgh (GB)
- ORCID
- Heafield, Kenneth
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 440-445
- OpenAlex
- W2606891064
- Cited by
- 585
- References
- 0
- Citations by year
- 2026: 57 · 2025: 89 · 2024: 104 · 2023: 88 · 2022: 79 · 2021: 83 · 2020: 54 · 2019: 24 · 2018: 7
- FWCI
- 10.60
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Nvidia
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- sparse communication, distributed gradient descent, stochastic gradient descent, gradient compression, quantization, neural machine translation, convergence rate, communication efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.