Sparse Communication for Distributed Gradient Descent
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates.Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices.This method can be combined with quantization to further improve the compression.We explore different configurations and apply them to neural machine translation and MNIST image classification tasks.Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task.Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sparsification, on the other hand, proposes to only communicate the informative elements of the gradient or model vectors among nodes [128], [129].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [129]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tolu Alabi, Jeffrey D Blanchard, Bradley Gordon, and Russel Steinbach. 2012. Fast k-selection algorithms for graphics processing units. Journal of Experi- mental Algorithmics (JEA) 17:4-2.
- Dan Alistarh, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2016. QSGD: randomized quan- tization for communication-optimal stochas- tic gradient descent. CoRR abs/1610.02132. http://arxiv.org/abs/1610.02132. · arXiv
- Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project adam: Building an efficient and scalable deep learning training system. In OSDI. volume 14, pages 571- 582.
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems. pages 1223- 1231.
- Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016. Communication quantization for data-parallel training of deep neural networks. In Proceedings of the Workshop on Machine Learn- ing in High Performance Computing Environments. IEEE Press, pages 1-8.
- Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In Program of the 13th International Workshop on Spoken Language Translation (IWSLT 2016).
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
- J. Lei Ba, J. R. Kiros, and G. E. Hinton. 2016. Layer Normalization. ArXiv e-prints .
- Ramón P Ñeco and Mikel L Forcada. 1996. Beyond mealy machines: Learning translators with recurrent neural networks. In Proceedings of the 1996 Inter- national Neural Network Society Annual Meeting. San Diego, California, USA.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings 40th Annual Meeting of the Association for Com- putational Linguistics. Philadelphia, PA, pages 311- 318.
- Rajat Raina, Anand Madhavan, and Andrew Y Ng. 2009. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th an- nual international conference on machine learning. ACM, pages 873-880.
- Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient de- scent. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Sys- tems 24, Curran Associates, Inc., pages 693-701. http://papers.nips.cc/paper/4390-hogwild-a-lock- free-approach-to-parallelizing-stochastic-gradient- descent.pdf.
- Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1-bit stochastic gradi- ent descent and application to data-parallel distributed training of speech DNNs. In Interspeech. https://www.microsoft.com/en- us/research/publication/1-bit-stochastic-gradient- descent-and-application-to-data-parallel- distributed-training-of-speech-dnns/.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation sys- tems for WMT 16. In Proceedings of the ACL 2016 First Conference on Machine Translation (WMT16).
- Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. 2010. Parallelized stochastic gradient descent. In Advances in neural information process- ing systems. pages 2595-2603.
- Daniel Zwillinger and Stephen Kokoska. 1999. CRC standard probability and statistics tables and formu- lae. CRC Press.
Cited by
Items 601–700 of 783
Communication Optimization Schemes for Accelerating Distributed Deep Learning Systems
Lee, J., Choi, H., Jeong, H., Noh, B., & Shin, J. S. (2020) · Applied Sciences · doi:10.3390/app10248846 · Cited by 1
Network Update Compression for Federated Learning
Kathariya, B., Li, L., Li, Z., Duan, L., & Liu, S. (2020) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip49819.2020.9301815 · Cited by 1
Basics of Distributed Machine Learning
Jiang, J., Cui, B., & Zhang, C. (2021) · Big data management · doi:10.1007/978-981-16-3420-8_2 · Cited by 1
AFSD: Adaptive Feature Space Distillation for Distributed Deep Learning
Khaleghian, S., Ullah, H., Johnsen, E. B., Andersen, A., & Marinoni, A. (2022) · IEEE Access · doi:10.1109/access.2022.3197646 · Cited by 1
Zhang, N., & Tao, M. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3228813 · Cited by 1
Afrabandpey, H., Rangu, G., Zhang, H., Criri, F., Aksu, E., & Tavakoli, H. R. (2022) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip56404.2022.10008860 · Cited by 1
Communication-Efficient Adam-Type Algorithms for Distributed Data Mining
Xian, W., Huang, F., & Huang, H. (2022) · Proceedings · doi:10.1109/icdm54844.2022.00162 · Cited by 3
Xiao, D., Jiang, P., & Li, M. (2022) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys57074.2022.00236 · Cited by 2
Vertical asynchronous federated learning: algorithms and theoretic guarantees
Chen, T., Jin, X. P., Sun, Y., & Yin, W. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-319037-7.00020-x · Cited by 1
Energy Optimization for Hierarchical Federated Learning based on Over-the-Air Computing
Zheng, K., & Li, Z. (2023) · IEEE Smart World Congress (SWC) · doi:10.1109/swc57546.2023.10448654 · Cited by 1
Wu, T., Yu, W., Guo, M., Nie, W., Jia, Z., & Xin, L. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00375 · Cited by 1
Distributed optimization algorithm for multi‐agent networks with lazy gradient information
Mo, L., Yang, Y., & Huang, X. (2024) · Asian Journal of Control · doi:10.1002/asjc.3422 · Cited by 1
Minimal Communication-Cost Statistical Learning
Sefidgaran, M., Zaidi, A., & Krasnowski, P. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619653 · Cited by 1
Sparse Incremental Aggregation in Multi-Hop Federated Learning
Mukherjee, S., Razmi, N., Dekorsy, A., Popovski, P., & Matthiesen, B. (2024) · IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) · doi:10.1109/spawc60668.2024.10694443 · Cited by 1
DynSplit: A Dynamic Split Learning Scheme for 5G-Enpowered Metaverse
Shu, Y., Gu, P., Adjih, C., Chen, C. S., & Serhrouchni, A. (2024) · IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom) · doi:10.1109/metacom62920.2024.00043 · Cited by 1
Communication‐Efficient Distributed Gradient Descent via Random Projection
Qi, H., & Gao, Y. (2024) · Stat · doi:10.1002/sta4.70030 · Cited by 1
DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models
Tsouvalas, V., Asano, Y. M., & Saeed, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825398 · Cited by 2
Accelerating Federated Learning with Adaptive Test-Driven Quantization at Edge Network
Liu, Y., Fan, C., & Zhang, X. (2024) · IEEE International Conference on Communication Technology (ICCT) · doi:10.1109/icct62411.2024.10946551 · Cited by 1
Priority-based parameter propagation for distributed deep neural network training
Jayarajan, A. (2019) · cIRcle (University of British Columbia) · doi:10.14288/1.0380523 · Cited by 1
Sparsification as a Remedy for Staleness in Distributed Asynchronous SGDPreprint
Candela, R., Franzese, G., Filippone, M., & Pietro Michiardi, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09466 · Cited by 1
Gradient Sparification for Asynchronous Distributed TrainingPreprint
Yan, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10929 · Cited by 1
DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed AveragingPreprint
Zhou, Q., Zhang, Y., Li, P., Liu, X., Yang, J., Wang, R., & Huang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00441 · Cited by 1
Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML TrainingPreprint
Tan, W., Shi, X., Lv, C., & Zhao, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05330 · Cited by 1
Private Federated Learning with Autotuned CompressionPreprint
Ullah, E., Choquette-Choo, C. A., Kairouz, P., & Oh, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10999 · Cited by 9
SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign DecodingPreprint
Park, C., & Lee, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01340 · Cited by 4
Fair Allocation of Bandwidth At Edge Servers For Concurrent Hierarchical Federated LearningPreprint
Hossen, M. A., Siddika, F., & Zhang, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.04921 · Cited by 1
Towards Scalable and Secure Federated Learning: Key Issues and Future ResearchPreprint
Daniels, M., Gallus, S., & Wood, R. (2025) · doi:10.36227/techrxiv.173894842.25024919/v1 · Cited by 1
Sinaga, K. P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.16101 · Cited by 1
Optimizing Frequent Checkpointing via Low-Cost Differential for Distributed Training SystemsPreprint
Yao, C., Hu, Y., Liu, F., Liu, Z., Wang, L., Li, M., & Feng, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.04084 · Cited by 1
Trading-off Accuracy and Communication Cost in Federated LearningPreprint
Villani, M. J., Natale, E., & Mallmann-Trenn, F. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.14246 · Cited by 1
Li, K., Wu, Y., Tian, J., Tian, W., & Ye, Z. (2019) · International Conference on Information Technology: IoT and Smart City (ICIT) · doi:10.1145/3377170.3377245
Hylo: Hybrid Layer-Based Optimization to Reduce Communication in Distributed Deep Learning
Jiang, W., Peng, J., Liu, P., & Jin, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60259-8_50
Pace Control via Adaptive Dropout for Federated Training: A Work-in-Progress Report
Wang, F., Shang, X., Shan, J., & Ding, X. (2020) · IEEE Cloud Summit · doi:10.1109/ieeecloudsummit48914.2020.00036
Fast Training of POI Recommendation Models Using Gradient Compression
Sun, H., Wang, Y., Sun, J., & Sun, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69873-7_6
Courier: Real-Time Optimal Batch Size Prediction for Latency SLOs in BigDL
Martínez, D. A., Bobde, S., Motyka, T., & Chen, L. Y. (2021) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3427921.3450233
Efficient Distributed Stochastic Gradient Descent Through Gaussian Averaging
Hu, K., Wu, C., & Zhu, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78609-0_4
Rate distortion comparison of a few gradient quantizersPreprint
Adikari, T. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09899
Large-scale training of deep neural networks
Dryden, N. (2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Distributed Gradient Optimization Algorithms
Jiang, J., Cui, B., & Zhang, C. (2021) · Big data management · doi:10.1007/978-981-16-3420-8_3
Alistarh, D., Markov, I., & Nadiradze, G. (2022) · ACM SIGACT News · doi:10.1145/3544979.3544991
Quantized Adaptive Subgradient Algorithms and Their ApplicationsPreprint
Xu, K., Wangni, J., Zhang, Y., Ye, D., Wu, J., & Zhao, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05631
Leveraging structural properties for large-scale optimization
Moharrer, A. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20410362
Privacy-enhancing mechanisms for securing data access and analysis
Guo, J. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/161735 · Cited by 1
Downlink Compression Improves TopK SparsificationPreprint
Zou, W., Sterck, H. D., & Liu, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15203
Dynamic Strategies for High Performance Training of Knowledge Graph Embeddings
Panda, A., & Vadhiyar, S. S. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545075
Sequential Decision Making for Large Scale Machine Learning
Liu, R. (2023) · Deep Blue (University of Michigan) · doi:10.7302/7259
Deep Hierarchy Quantization Compression algorithm based on Dynamic Sampling
Jiang, W., & Liu, G. (2023) · International Conference on Electronic Communication and Artificial Intelligence (ICECAI) · doi:10.1109/icecai58670.2023.10176920
A Theoretical Study on Artificial Intelligence Training
Han, D., & Yoo, H. (2023) · Springer eBooks · doi:10.1007/978-3-031-34237-0_2
Li, G., Liao, M., Chao, Y., & Lv, P. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191729
Tan, J., Yao, C., & Guo, Z. (2023) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3600061.3603140
Locally Accumulated Adam For Distributed Training With Sparse Updates
Chen, Y., & Deligiannis, N. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222032
Distributed Adaptive Optimization with Divisible Communication
Xu, A. P., & Bai, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43418-1_39
Augmenting Distributed AI Training with Loss-tolerant Transmission
Chen, Z., Shi, L., Gao, Y., Liu, X., Ai, X., Liu, S., & Xu, Y. (2023) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3603165.3607399
MSDU: Multi-step Delayed Communication for Efficient Distributed Deep Learning
Yao, F., Tan, B., Liu, B., & Ji, Z. (2023) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie59714.2023.10281239
Analysis and Optimization of Model Aggregation Performance in Federated Learning
Yang, H., Ding, T., & Yuan, X. (2023) · International Conference on Communications · doi:10.1109/icc45041.2023.10278909
Communication Efficient Distributed Learning
Singh, N., Data, D., & Diggavi, S. N. (2023) · Springer eBooks · doi:10.1007/978-3-031-40787-1_6 · Cited by 1
Seo, J., Rong, C., & Kim, M. (2025) · IEEE Access · doi:10.1109/access.2025.3648064
Fundamentals and Preliminaries of Federated Learning
Chen, M., & Cui, S. R. (2023) · Wireless networks · doi:10.1007/978-3-031-51266-7_2
A Stochastic Gradient Approach for Communication Efficient Confederated Learning
Wang, B., Fang, J., Li, H., & Eldar, Y. C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448035
Contributions to Federated Learning and First-Order Optimization
Dieuleveut, A. (2023) · HAL
Permutation mask: a combined gradient sparsification for federated learning
Zhou, S., & Zhou, Y. (2024) · Journal of nonparametric statistics · doi:10.1080/10485252.2024.2370874
Membership Privacy Protection for Federated Learning in Web 3.0
Shen, M., Tang, X., Wang, W., & Zhu, L. (2024) · Digital Privacy and Security · doi:10.1007/978-981-97-5752-7_4
SGC: Similarity-Guided Gradient Compression for Distributed Deep Learning
Liu, J., Huang, J., Li, Y., Li, Z., Lyu, W., Jiang, W., & Wang, J. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682863
Stochastic Biased Gradient Methods
Hu, Y. (2024) · Springer eBooks · doi:10.1007/978-3-030-54621-2_887-1
How Useful is Communication Scheduling for Distributed Training?
Zhao, Y., Liu, X., & Jin, X. (2024) · Modern Network technologies · doi:10.1109/monetec60984.2024.10768125
On Quantifying the Gradient Inversion Risk of Data Reuse in Federated Learning Systems
Huang, J., Chen, L. Y., & Roos, S. (2024) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds64841.2024.00031
HJCFL: Hashcash and Jaya-Based Communication Efficient Federated Learning
Mondal, S., & Jana, P. K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-81404-4_20
Navigating the Frontiers of Federated Learning: Opportunities and ObstaclesPreprint
Glanville, C., Gallus, S., & Kash, C. (2025) · HAL
Research on Automatic Recognition Algorithm of Case Images Based on Convolutional Neural Network
Li, Y., Li, K.-Y., & Li, Q. (2024) · IEEE International Conference on Data Science and Computer Application (ICDSCA) · doi:10.1109/icdsca63855.2024.10859931
Coding-based In-Network Aggregation for Distributed Deep Learning
Chang, K.-C., Kuo, Y.-C., & Lin, K. C. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901081
Second-Order Wireless Federated Leaning via Nonparametric Hessian Estimation
Hamidi, S. M., & Bereyhi, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889600
Optimizing Training Performance via Omni-Block Sparsity Adaptation
Li, W. (2024) · International Conference on Communication Technology and Information Technology (ICCTIT) · doi:10.1109/icctit64404.2024.10928544
Data Privacy-Preserving and Communication Efficient Federated Multilinear Compressed Learning
Xiao, D., Yang, Z., Zhang, M., & Chen, L. (2024) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom63139.2024.00203
Performance Difference Based Lazy Aggregation in Federated Learning Human Activity Recognition
Zhang, Q., & Liu, Y. (2024) · International Conference on Ubiquitous Computing and Communications (IUCC) · doi:10.1109/iucc65928.2024.00029
Dynamic Single-Step Synthetic Features Compression for Communication-Efficient Federated Learning
Liu, X., Zhang, Y., Pang, Z., Wu, Z., Bai, H., & Zhang, Y. (2025) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce65916.2025.11065911
A Communication-Efficient Federated Contrastive Learning Framework for Intrusion Detection
Guo, C., Cheng, J., Yang, S., & Zhang, D. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9958-2_24
Optimization, Communication, and Personalization in Federated Learning for Massive NetworksPreprint
Gallus, S., Mercer, A., Singh, P., & Cho, D. (2025) · Preprints.org · doi:10.20944/preprints202507.1037.v1 · Cited by 2
FedLOC:A Layer Output Based Compression Algorithm for Federated Learning
Ouyang, P., Xiao, D., Zhou, J., & Wu, W. (2024) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc64274.2024.00098
Accelerating Distributed DNN Training Efficiency via Layered Gradient Clustering
Yin, Y., Huang, J., Lu, S., Zhou, Q., Jiang, J., & Liu, W. (2025) · International Conference on Communication Engineering and Technology (ICCET) · doi:10.1109/iccet65872.2025.11082610
Personalized and Robust Federated Learning in Heterogeneous EnvironmentsPreprint
Johnson, A. E., Lee, B., Smith, C., Kim, D., Martínez, E., Noor, F., Chen, G., Park, H., & Gallus, S. (2025) · doi:10.36227/techrxiv.175355119.90288089/v1
Unbiased Data-Driven Dynamic Threshold Sparsification for Communication-Efficient Federated Learning
Yu, J., Shu, N., Wu, T., Wang, H., Jiang, R., & Chang, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9805-9_19
Saving Memory via Residual Reduction for DNN Training with Compressed Communication
Zheng, X., Ming, Z., Hu, Y., Yao, C., Zhou, W., Wang, R., Chen, X., & Feng, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_16
Secure distributed machine learning for edge computing
Plug, R. B. F., Amare, S. Y., Jati, P. H. P., Kievit, R., Kawu, A. A., Liang, Y., & van Reisen, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-34109-0.00001-2
FDA-Opt: Federated Fine-Tuning via Dynamic Update Schedules
Theologitis, M., Samoladas, V., & Deligiannakis, A. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840785
Efficient Quantization Mean Estimation for Distributed Learning
Mao, X., Wang, H., & Zhang, X. (2025) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2025.2572324
Weighted Probabilistic Mask Aggregation for Fault Tolerant Federated Learning
Song, R., Jin, R., Jiang, S., Huang, C., & Liu, J. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3621158
Adaptive Gradient Quantization with Bit Allocation for Distributed Deep Learning
Gao, F., Yan, X., Jin, J., Yang, W., Duan, L., & Chen, Z. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209773
LMC: Lightweight Message Collection for Distributed Training of Deep Learning Models
Zhang, Y., Wang, Y., Jia, J., & Liu, Y. (2025) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc67675.2025.00073
FedSHE–CQ: A communication-efficient homomorphic encryption framework for federated learning
Wei, X., Huang, R., Lei, L., & Wu, J. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111813
Descours, A., Deroose, L., & Ramon, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.05593
It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMsPreprint
Wu, J., Huang, P., Wen, J., & Han, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.00486
MultiGS: Multi-Dimensional Information-Aware Gradient Sparsification
Ma, J., Donglai, L., Xie, W., Li, H., Li, Y., & Fang, L. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3636885
Li, X., Sun, J., Yang, Z., & Ding, K. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3637373 · Cited by 1
Zhao, Z., Chen, Q.-F., Li, J., Fan, D., & Chakrabarti, C. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261806
A Study on Gradient Sparsity-based Federated Learning Training Method Utilizing DPU
Li, Y., Gao, P., Dong, Z., Xu, D. S., Zhang, Z., & Zhang, R. (2025) · International Conference on Big Data Engineering and Education (BDEE) · doi:10.1109/bdee67464.2025.00012
Optimization Methods and Software for Federated Learning
Burlachenko, K. (2025) · arXiv (Cornell University) · doi:10.25781/kaust-3tj71
Efficient Optimization Algorithms for Nonconvex Machine Learning Problems
Xian, W. (2024) · University Libraries (University of Maryland) · doi:10.13016/r6ko-uppv
Enhanced Architectures and Optimization Methods for Efficient Language Modeling
Mohtashami, A. (2025) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-10940
Yi, K. (2025) · arXiv (Cornell University) · doi:10.25781/kaust-80584
Haroush, M., & Soudry, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.18313
Metadata
- Authors
- Aji, A. F., & Heafield, K.
- Affiliations
- University of Edinburgh (GB)
- ORCID
- Heafield, Kenneth
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 440-445
- OpenAlex
- W2606891064
- Cited by
- 585
- References
- 0
- Citations by year
- 2026: 57 · 2025: 89 · 2024: 104 · 2023: 88 · 2022: 79 · 2021: 83 · 2020: 54 · 2019: 24 · 2018: 7
- FWCI
- 10.60
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Nvidia
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- sparse communication, distributed gradient descent, stochastic gradient descent, gradient compression, quantization, neural machine translation, convergence rate, communication efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.