ZeRO-Offload: Democratizing Billion-Scale Model Training
Open accessAbstract
Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Offloading allows running LLMs with a single low-end accelerator by loading parameters from RAM just-in-time for each forward pass.
“This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass.”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2023) · cites this paper as Ren et al., 2021
Section 4.1 (Single GPU Schedule) says ZeRO-Offload keeps fp16 parameters on the GPU, so forward propagation requires no CPU communication. Although it enables large-model training on a single GPU, it does not load parameters from RAM just in time for each forward pass; the paper describes that layer-by-layer transfer as L2L's approach in Section 2. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,189 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 3× · see citing works
“Recent studies in fact show that larger models are more resource-efficient to train than smaller ones [12] for a given accuracy target.”
BackgroundMethodInfluentialGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180 · see citing works
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124 · see citing works
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · see citing works
Self-citationMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Shen, L., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J. S., Vaughan, B., Damania, P., et al. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415530 · Cited by 684 · see citing works
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561 · see citing works
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N. E., Clemons, J., Zulfiqar, A., & Keckler, S. W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783721 · Cited by 416 · see citing works
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Huang, C.-C., Jin, G., & Li, J. (2020) · Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3373376.3378530 · Cited by 158 · cited 2× · see citing works
“The third approach uses an external memory such as the CPU memory as an extension of GPU memory to increase memory capacity during training [8, 9, 11, 17, 23, 24, 33].”
BackgroundMethodPipeDream: Fast and Efficient Pipeline Parallel DNN Training
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03377 · Cited by 153 · cited 2× · see citing works
“Two prominent examples of scale out training is model parallelism [5, 28] and pipeline parallelism [7,10], both partitioning the model states and the residual states across multiple GPUs.”
BackgroundMethodPeng, X., Shi, X., Dai, H., Jin, H., Ma, W., Xiong, Q., Yang, F., & Qian, X. (2020) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3373376.3378505 · Cited by 142 · see citing works
Mitra, G., Johnston, B., Rendell, A. P. L., McCreath, E. C., & Zhou, J. W. (2013) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2013.207 · Cited by 92 · see citing works
AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming
Hildebrand, M., Khan, J. A., Trika, S., Lowe-Power, J., & Akella, V. (2020) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3373376.3378465 · Cited by 69 · see citing works
Ren, J., Luo, J., Wu, K., Zhang, M., Jeon, H., & Li, D. (2021) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00057 · Cited by 61 · see citing works
Self-citationBuddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs
Choukse, E., Sullivan, M. B., O’Connor, M., Erez, M., Pool, J., Nellans, D. W., & Keckler, S. W. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00080 · Cited by 60 · see citing works
Jin, H., Liu, B., Jiang, W., Ma, Y., Shi, X., He, B., & Zhao, S. (2018) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3243904 · Cited by 33 · see citing works
Efficient Memory Management for GPU-based Deep Learning SystemsPreprint
Zhang, J., Yeung, S. H., Shu, Y., He, B., & Wang, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.06631 · Cited by 28 · see citing works
Pretrained Language Models for Dialogue Generation with Multiple Input Sources
Cao, Y. H., Bi, W., Fang, M., & Tao, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.81 · Cited by 25 · see citing works
Training Large Neural Networks with Constant Memory using a New Execution Algorithm
Pudipeddi, B., Mesmakhosroshahi, M., Xi, J., & Bharadwaj, S. (2020) · arXiv · doi:10.48550/arxiv.2002.05645 · Cited by 25 · see citing works
The performance impact analysis of loop unrolling
Velkoski, G., Gušev, M., & Ristov, S. (2014) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro.2014.6859582 · Cited by 10 · see citing works
Flagged first, then by citations
2 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- [17][17] Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS '20, page 891-905, New York, NY, USA, 2020. Association for Computing Machinery. -
- [19][19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [21][21] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , 2020. -
- [26][26] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism on Neural Network Training. Journal of Machine Learning Research , 20, 2019. -
- [27][27] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Meshtensorflow: Deep learning for supercomputers. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 10435-10444, 2018. -
- [30][30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. -
- [33][33] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , PPoPP '18, page 41-53, New York, NY, USA, 2018. Association for Computing Machinery. -
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“Many other works aim to increase of the scale of models, while limiting communication overheads (Rajbhandari et al., 2020; Lepikhin et al., 2020; Li et al., 2020; Rasley et al., 2020; Rajbhandari et al., 2021; Ren et al., 2021; Narayanan et al., 2021a).”
BackgroundEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Efficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639 · see citing works
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436 · see citing works
(2021) · doi:10.1145/3458817.3476205 · Cited by 257 · see citing works
Self-citationA Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 334 · see citing works
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 165 · see citing works
Li, S., & Hoefler, T. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476145 · Cited by 124 · see citing works
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
An overview of large AI models and their applications
Tu, X., He, Z., Huang, Y., Zhang, Z., Yang, M., & Zhao, J. C. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00065-8 · Cited by 95 · see citing works
Won, W., Heo, T., Rashidi, S., Sridharan, S., Srinivasan, S., & Krishna, T. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00035 · Cited by 88 · see citing works
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
TCMChat: A generative large language model for traditional Chinese medicine
Dai, Y., Shao, X., Zhang, J., Chen, Y., Chen, Q., Liao, J., Chi, F., Zhang, J., & Fan, X. (2024) · Pharmacological Research · doi:10.1016/j.phrs.2024.107530 · Cited by 75 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Lai, Z., Li, S., Tang, X., Ge, K., Liu, W., Duan, Y., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247001 · Cited by 69 · see citing works
Self-citationPre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., & Yang, M. F. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00078 · Cited by 61 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
HPC-Coder: Modeling Parallel Programs using Large Language Models
Nichols, D., Marathe, A., Menon, H., Gamblin, T., & Bhatele, A. S. (2024) · ISC High Performance (ISC) · doi:10.23919/isc.2024.10528929 · Cited by 59 · see citing works
Sequence Parallelism: Long Sequence Training from System Perspective
Li, S., Xue, F., Baranwal, C., Li, Y., & You, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.134 · Cited by 88 · see citing works
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Li, K., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00686 · Cited by 47 · see citing works
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language ModelsPreprint
Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D. X., & Stoica, I. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07988 · Cited by 51 · see citing works
Betty: Enabling Large-Scale GNN Training with Batch-Level Graph Partitioning
Yang, S., Zhang, M., Dong, W., & Li, D. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575725 · Cited by 47 · see citing works
Self-citationHanayo: Harnessing Wave-Like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Liu, Z.-M., Cheng, S., Zhou, H., & You, Y. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3581784.3607073 · Cited by 73 · see citing works
Prompt Consistency for Zero-Shot Task Generalization
Zhou, C., He, J., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.192 · Cited by 45 · see citing works
An Efficient 2D Method for Training Super-Large Deep Learning Models
Xu, Q., & You, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00031 · Cited by 45 · see citing works
Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management
Fang, J., Zhu, Z., Li, S., Su, H., Yu, Y., Zhou, J., & You, Y. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3219819 · Cited by 39 · see citing works
Achieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 39 · see citing works
Efficient Large-Scale Language Model Training on GPU ClustersPreprint
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M. A., Korthikanti, V. A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · arXiv (Cornell University) · Cited by 35 · see citing works
Distributed Training of Large Language Models
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00126 · Cited by 34 · see citing works
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
Wang, Z., Xu, H., Yang, X., Jiang, Z., & Liu, J. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109490 · Cited by 31 · see citing works
Guo, C., Zhang, R., Xu, J., Leng, J., Liu, Z., Huang, Z., Guo, M., Wu, H., Zhao, S., Zhao, J., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640423 · Cited by 32 · see citing works
Greener, J. G., & Jones, D. T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0256990 · Cited by 29 · see citing works
AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Sun, Z., Cao, H., Wang, Y., Feng, G., Chen, S., Wang, H., & Chen, W. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651359 · Cited by 29 · see citing works
Characterizing Multi-Instance GPU for Machine Learning Workloads
Li, B., Gadepally, V., Samsi, S., & Tiwari, D. (2022) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw55747.2022.00124 · Cited by 29 · see citing works
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Jang, H., Song, J., Jung, J., Park, J., Kim, Y., & Lee, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00034 · Cited by 28 · see citing works
Lu, Y., Liu, D., Liang, Z., Liu, R., Chen, P., Liu, Y., Li, J., Feng, Z., Li, L. M., Sheng, B., et al. (2025) · National Science Review · doi:10.1093/nsr/nwaf039 · Cited by 31 · see citing works
TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Nie, X., Miao, X., Yang, Z. C., & Cui, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00241 · Cited by 26 · see citing works
HongTu: Scalable Full-Graph GNN Training on Multiple GPUs
Wang, Q., Chen, Y., Wong, W., & He, B. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3626733 · Cited by 25 · see citing works
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 25 · see citing works
PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters
Zhang, J., Niu, G., Dai, Q., Li, H., Wu, Z., Dong, F., & Wu, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126661 · Cited by 23 · see citing works
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 22 · see citing works
JUWELS Booster – A Supercomputer for Large-Scale AI Research
Kesselheim, S., Herten, A., Krajsek, K., Ebert, J., Jitsev, J., Cherti, M., Langguth, M., Gong, B., Stadtler, S., Mozaffari, A., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90539-2_31 · Cited by 21 · see citing works
Tale of Two Cs: Computation vs. Communication Scaling for Future Transformers on Future Hardware
Pati, S., Aga, S., Islam, M., Jayasena, N., & Sinclair, M. D. (2023) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc59245.2023.00026 · Cited by 21 · see citing works
Training and Serving System of Foundation Models: A Comprehensive Survey
Zhou, J., Chen, Y., Hong, Z., Chen, W., Yu, Y., Zhang, T., Wang, H., Zhang, C., & Zheng, Z. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3380828 · Cited by 20 · see citing works
Liu, J., Li, D., Gioiosa, R., & Li, J. (2021) · International Conference on Supercomputing · doi:10.1145/3447818.3460355 · Cited by 20 · see citing works
Self-citationA Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training
Li, S., Lü, K., Lai, Z., Liu, W., Ge, K., & Li, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3406420 · Cited by 17 · see citing works
Self-citationBang, J., Choi, Y., Kim, M., Kim, Y., & Rhu, M. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00021 · Cited by 17 · see citing works
Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link
Xu, D., Feng, Y., Shin, K.-S., Kim, D., Jeon, H., & Li, D. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00100 · Cited by 17 · see citing works
Self-citationHPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training
Duan, Y., Lai, Z., Li, S., Liu, W., Ge, K., Liang, P., & Li, D. (2022) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster51413.2022.00043 · Cited by 17 · see citing works
Self-citationSTRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training
Sun, X., Wang, W., Qiu, S., Yang, R., Huang, S., Xu, J., & Wang, Z. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00076 · Cited by 17 · see citing works
Li, F., Zhao, S., Qing, Y., Chen, X., Guan, X., Wang, S., Zhang, G., & Cui, H. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247883 · Cited by 17 · see citing works
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter PretrainingPreprint
Lin, J., An, Y., Bai, J., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Li, Y., Lin, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03888 · Cited by 23 · see citing works
Performance Characterization of CXL Memory and Its Use Cases
Wang, X., Liu, J., Wu, J., Yang, S., Ren, J., Shankar, B., & Li, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00097 · Cited by 16 · see citing works
Self-citationProteus: Simulating the Performance of Distributed DNN Training
Duan, J., Li, X., Xu, P., Zhang, X., Yan, S., Liang, Y. E., & Lin, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3443255 · Cited by 15 · see citing works
Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly
Woisetschläger, H., Erben, A., Wang, S., Mayer, R., & Jacobsen, H. (2024) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3650203.3663331 · Cited by 20 · see citing works
RAMP: A flat nanosecond optical network and MPI operations for distributed deep learning systems
Ottino, A., Benjamin, J., & Zervas, G. (2023) · Optical Switching and Networking · doi:10.1016/j.osn.2023.100761 · Cited by 14 · see citing works
FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation
Kim, T., Park, C., Mukimbekov, M., Hong, H., Kim, M., Jin, Z., Kim, C., Shin, J.-Y., & Jeon, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636238 · Cited by 14 · see citing works
MAGIS: Memory Optimization via Coordinated Graph Transformation and Scheduling for DNN
Chen, R., Ding, Z., Zheng, S., Zhang, C., Leng, J., Liu, X., & Liang, Y. E. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651330 · Cited by 14 · see citing works
Han, H., Zheng, X., Wen, Y., Hao, Y., Feng, E., Liang, L., Mu, J., Li, X., Ma, T., Jin, P., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674168 · Cited by 13 · see citing works
AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning
Singh, S., & Bhatele, A. S. (2022) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps53621.2022.00065 · Cited by 13 · see citing works
M6-T: Exploring Sparse Expert Models and BeyondPreprint
An, Y., Lin, J., Men, R., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Wang, J., Li, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15082 · Cited by 35 · see citing works
A Cost-Efficient Failure-Tolerant Scheme for Distributed DNN Training
Chen, M., Hua, Y., Bai, R., & Huang, J. (2023) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd58817.2023.00031 · Cited by 12 · see citing works
Enabling Large Dynamic Neural Network Training with Learning-based Memory Management
Ren, J., Xu, D., Yang, S., Zhao, J., Li, Z., Navasca, C., Wang, C., Xu, H., & Li, D. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00066 · Cited by 13 · see citing works
Self-citationMaurya, A., Ye, J., Rafique, M. M., Cappello, F., & Nicolae, B. (2024) · doi:10.1145/3659995.3660038 · Cited by 11 · see citing works
PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption
Tan, Y., Tan, C. L., Mi, Z., & Chen, H. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707224 · Cited by 12 · see citing works
Distributed training of large language models: A survey
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100174 · Cited by 12 · see citing works
Momentum-driven adaptive synchronization model for distributed DNN training on HPC clusters
Zhang, Z., Ji, Z., & Wang, C. (2021) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2021.09.007 · Cited by 11 · see citing works
CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs
Sun, Q., Liu, Y., Yang, H., Zhang, R., Dun, M., Li, M., Liu, X., Xiao, W., Li, Y., Luan, Z., et al. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00044 · Cited by 11 · see citing works
InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models
Nagrecha, K., Liu, L., Delgado, P., & Padmanabhan, P. (2023) · ACM Conference on Recommender Systems · doi:10.1145/3604915.3608778 · Cited by 10 · see citing works
Deep Optimizer States: Towards Scalable Training of Transformer Models using Interleaved Offloading
Maurya, A., Ye, J., Rafique, M. M., Cappello, F., & Nicolae, B. (2024) · International Middleware Conference · doi:10.1145/3652892.3700781 · Cited by 10 · see citing works
Buffalo: Enabling Large-Scale GNN Training via Memory-Efficient Bucketization
Yang, S., Zhang, M., & Li, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00083 · Cited by 10 · see citing works
Self-citationMixPipe: Efficient Bidirectional Pipeline Parallelism for Training Large-Scale Models
Zhang, W., Zhou, B., Tang, X., Zhaoxing, W., & Hu, S. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247730 · Cited by 10 · see citing works
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed TrainingPreprint
Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I. G., Mahoney, M. W., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14129 · Cited by 11 · see citing works
Breaking Barriers: Expanding GPU Memory with Sub-Two Digit Nanosecond Latency CXL Controller
Gouk, D., Kang, S.-K., Bae, H., Ryu, E., Lee, S., Kim, D., Jang, J., & Jung, M. (2024) · USENIX Workshop on Hot Topics in Storage and File Systems · doi:10.1145/3655038.3665953 · Cited by 12 · see citing works
Implementing CUDA Unified Memory in the PyTorch Framework
Choi, J., Yeom, H. Y., & Kim, Y. (2021) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C) · doi:10.1109/acsos-c52956.2021.00029 · Cited by 9 · see citing works
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model TrainingPreprint
Karakus, C., Huilgol, R., Wu, F. Q., Subramanian, A., Daniel, C., Çavdar, D., Xu, T., Chen, H., Rahnama, A., & Quintela, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05972 · Cited by 17 · see citing works
Sparse Fusion for Multimodal TransformersPreprint
Ding, Y., Rich, A., Wang, M. L., Stier, N., Turk, M., Sen, P., & Höllerer, T. (2022) · doi:10.31219/osf.io/f7jvn · Cited by 9 · see citing works
A Survey on Performance Modeling and Prediction for Distributed DNN Training
Guo, Z., Tang, Y., Zhai, J., Yuan, T., Jin, J., Li, W., Zhao, Y., & Li, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3476390 · Cited by 8 · see citing works
Machine Learning-Guided Memory Optimization for DLRM Inference on Tiered Memory
Ren, J., Ma, B., Yang, S., Francis, B., Ardestani, E. K., Si, M., & Dong, L. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00121 · Cited by 9 · see citing works
Self-citationGKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Tan, S., Tam, W. L., Wang, Y., Gong, W., Zhao, S., Zhang, P., & Tang, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.15 · Cited by 8 · see citing works
Adaptive Sparse Deep Neural Network Inference on Resource-Constrained Cost-Efficient GPUs
Dun, M., Zhang, X., Cao, H., Zhang, Y., Huang, J., & Ye, X. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363545 · Cited by 8 · see citing works
Watching a Language Model Learning Chess
Stöckl, A. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_153 · Cited by 8 · see citing works
BMCook: A Task-agnostic Compression Toolkit for Big Models
Zhang, Z., Gong, B., Chen, Y., Han, X., Zeng, G., Zhao, W., Chen, Y., Liu, Z., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.40 · Cited by 8 · see citing works
Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Lin, Z., Miao, Y., Xu, G., Li, C., Saarikivi, O., Maleki, S., & Yang, F. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00067 · Cited by 7 · see citing works
Efficient inference, training, and fine-tuning of protein language models
Çelik, M. H., & Xie, X. (2025) · iScience · doi:10.1016/j.isci.2025.113495 · Cited by 7 · see citing works
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Jiang, Y., Fu, F., Miao, X., Nie, X., & Cui, B. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/238 · Cited by 7 · see citing works
Bandwidth Characterization of DeepSpeed on Distributed Large Language Model Training
Hanindhito, B., Patel, B., & John, L. K. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00031 · Cited by 8 · see citing works
APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes
Wei, Y., Du, J., Jiang, J., Shi, X., Zhang, X., Huang, D., Xiao, N., & Lu, Y. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00096 · Cited by 7 · see citing works
pommDNN: Performance optimal GPU memory management for deep neural network training
Chen, W., Dong, X., Chen, X. F., Liu, S., Qin, X., & Wang, Q. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.10.025 · Cited by 7 · see citing works
RT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference
Kang, W., Lee, J., Lee, Y., Oh, S., Lee, K., & Chwa, H. S. (2024) · IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) · doi:10.1109/rtas61025.2024.00037 · Cited by 6 · see citing works
Survey on Efficient Training of Large Neural Networks
Gusak, J., Cherniuk, D., Shilova, A., Katrutsa, A., Bershatsky, D., Zhao, X., Eyraud‐Dubois, L., Shliazhko, O., Dimitrov, D., Oseledets, I. V., et al. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/769 · Cited by 6 · see citing works
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Anthony, Q. G., Awan, A. A., Rasley, J., He, Y., Shafi, A., Abduljabbar, M., Subramoni, H., & Panda, D. K. D. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00103 · Cited by 6 · see citing works
Self-citationA Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 8 · see citing works
Kanakarajan, K. R., & Sankarasubbu, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.137 · Cited by 7 · see citing works
SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers
Xu, D., Xu, M., Lou, C., Zhang, L. L., Huang, G., Jin, X., & Liu, X. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3617232.3624847 · Cited by 6 · see citing works
Scalable and Load-Balanced Full-Graph GNN Training on Multiple GPUs
Wang, Q., Chen, Y., Wong, W., & He, B. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3558641 · Cited by 5 · see citing works
Long-Term Ad Memorability: Understanding & Generating Memorable Ads
Harini, S. I., Singh, S., Singla, Y. K., Bhattacharyya, A., Baths, V., Chen, C., Shah, R. R., & Krishnamurthy, B. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00557 · Cited by 5 · see citing works
- Authors
- Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y.
- Affiliations
- University of California, Merced (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2101.06840
- OpenAlex
- W3121562065
- Cited by
- 151
- References
- 27
- Citations by year
- 2025: 20 · 2024: 46 · 2023: 44 · 2022: 16 · 2021: 23 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- large-scale model training, GPU memory efficiency, CPU offloading, data movement reduction, single-GPU training, model parallelism, distributed training, PyTorch
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
