The DOI matches its DataCite record; 26 of the 27 references it cites were found in a registry or an index.
ZeRO-Offload: Democratizing Billion-Scale Model Training
Open accessAbstract
Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass.”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2023) · cites this paper as Ren et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Swapping [23,42,55], recomputation [7,24] and their combination [40] have been utilized to reduce the peak memory of training.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [40]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 3×
“Recent studies in fact show that larger models are more resource-efficient to train than smaller ones [12] for a given accuracy target.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
Shen, L., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J. S., Vaughan, B., Damania, P., et al. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415530 · Cited by 996
Research works printed but not linked: 7
- [17][17] Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS '20, page 891-905, New York, NY, USA, 2020. Association for Computing Machinery. -
- [19][19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [21][21] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , 2020. -
- [26][26] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism on Neural Network Training. Journal of Machine Learning Research , 20, 2019. -
- [27][27] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Meshtensorflow: Deep learning for supercomputers. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 10435-10444, 2018. -
- [30][30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. -
- [33][33] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , PPoPP '18, page 41-53, New York, NY, USA, 2018. Association for Computing Machinery. -
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Many other works aim to increase of the scale of models, while limiting communication overheads (Rajbhandari et al., 2020; Lepikhin et al., 2020; Li et al., 2020; Rasley et al., 2020; Rajbhandari et al., 2021; Ren et al., 2021; Narayanan et al., 2021a).”
BackgroundEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Efficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
(2021) · doi:10.1145/3458817.3476205 · Cited by 257
Self-citationA Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 560
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 191
Li, S., & Hoefler, T. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476145 · Cited by 148
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
An overview of large AI models and their applications
Tu, X., He, Z., Huang, Y., Zhang, Z., Yang, M., & Zhao, J. C. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00065-8 · Cited by 106
Won, W., Heo, T., Rashidi, S., Sridharan, S., Srinivasan, S., & Krishna, T. (2023) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass57527.2023.00035 · Cited by 134
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
TCMChat: A generative large language model for traditional Chinese medicine
Dai, Y., Shao, X., Zhang, J., Chen, Y., Chen, Q., Liao, J., Chi, F., Zhang, J., & Fan, X. (2024) · Pharmacological Research · doi:10.1016/j.phrs.2024.107530 · Cited by 82
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Lai, Z., Li, S., Tang, X., Ge, K., Liu, W., Duan, Y., Qiao, L., & Li, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247001 · Cited by 75
Self-citationPre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., & Yang, M. F. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00078 · Cited by 102
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52
HPC-Coder: Modeling Parallel Programs using Large Language Models
Nichols, D., Marathe, A., Menon, H., Gamblin, T., & Bhatele, A. S. (2024) · ISC High Performance (ISC) · doi:10.23919/isc.2024.10528929 · Cited by 72
Sequence Parallelism: Long Sequence Training from System Perspective
Li, S., Xue, F., Baranwal, C., Li, Y., & You, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.134 · Cited by 181
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models
Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Li, K., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00686 · Cited by 74
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language ModelsPreprint
Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D. X., & Stoica, I. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07988 · Cited by 59
Betty: Enabling Large-Scale GNN Training with Batch-Level Graph Partitioning
Yang, S., Zhang, M., Dong, W., & Li, D. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575725 · Cited by 47
Self-citationHanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Liu, Z., Cheng, S., Zhou, H., & You, Y. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607073 · Cited by 56
Prompt Consistency for Zero-Shot Task Generalization
Zhou, C., He, J., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.192 · Cited by 68
An Efficient 2D Method for Training Super-Large Deep Learning Models
Xu, Q., & You, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00031 · Cited by 57
Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management
Fang, J., Zhu, Z., Li, S., Su, H., Yu, Y., Zhou, J., & You, Y. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3219819 · Cited by 46
Achieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 46
Efficient Large-Scale Language Model Training on GPU ClustersPreprint
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M. A., Korthikanti, V. A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · arXiv (Cornell University) · Cited by 35
Distributed Training of Large Language Models
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00126 · Cited by 34
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32
Wang, Z., Xu, H., Yang, X., Jiang, Z., & Liu, J. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109490 · Cited by 33
Guo, C., Zhang, R., Xu, J., Leng, J., Liu, Z., Huang, Z., Guo, M., Wu, H., Zhao, S., Zhao, J., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640423 · Cited by 44
Greener, J. G., & Jones, D. T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0256990 · Cited by 29
AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Sun, Z., Cao, H., Wang, Y., Feng, G., Chen, S., Wang, H., & Chen, W. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651359 · Cited by 45
Characterizing Multi-Instance GPU for Machine Learning Workloads
Li, B., Gadepally, V., Samsi, S., & Tiwari, D. (2022) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw55747.2022.00124 · Cited by 29
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Jang, H., Song, J., Jung, J., Park, J., Kim, Y., & Lee, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00034 · Cited by 30
Lu, Y., Liu, D., Liang, Z., Liu, R., Chen, P., Liu, Y., Li, J., Feng, Z., Li, L. M., Sheng, B., et al. (2025) · National Science Review · doi:10.1093/nsr/nwaf039 · Cited by 35
TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Nie, X., Miao, X., Yang, Z. C., & Cui, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00241 · Cited by 27
HongTu: Scalable Full-Graph GNN Training on Multiple GPUs
Wang, Q., Chen, Y., Wong, W., & He, B. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3626733 · Cited by 25
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 31
PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters
Zhang, J., Niu, G., Dai, Q., Li, H., Wu, Z., Dong, F., & Wu, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126661 · Cited by 23
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 23
JUWELS Booster – A Supercomputer for Large-Scale AI Research
Kesselheim, S., Herten, A., Krajsek, K., Ebert, J., Jitsev, J., Cherti, M., Langguth, M., Gong, B., Stadtler, S., Mozaffari, A., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90539-2_31 · Cited by 23
Tale of Two Cs: Computation vs. Communication Scaling for Future Transformers on Future Hardware
Pati, S., Aga, S., Islam, M., Jayasena, N., & Sinclair, M. D. (2023) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc59245.2023.00026 · Cited by 34
Training and Serving System of Foundation Models: A Comprehensive Survey
Zhou, J., Chen, Y., Hong, Z., Chen, W., Yu, Y., Zhang, T., Wang, H., Zhang, C., & Zheng, Z. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3380828 · Cited by 30
Liu, J., Li, D., Gioiosa, R., & Li, J. (2021) · International Conference on Supercomputing · doi:10.1145/3447818.3460355 · Cited by 20
Self-citationA Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training
Li, S., Lü, K., Lai, Z., Liu, W., Ge, K., & Li, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3406420 · Cited by 18
Self-citationBang, J., Choi, Y., Kim, M., Kim, Y., & Rhu, M. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00021 · Cited by 37
Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link
Xu, D., Feng, Y., Shin, K.-S., Kim, D., Jeon, H., & Li, D. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00100 · Cited by 18
Self-citationHPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training
Duan, Y., Lai, Z., Li, S., Liu, W., Ge, K., Liang, P., & Li, D. (2022) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster51413.2022.00043 · Cited by 17
Self-citationSTRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training
Sun, X., Wang, W., Qiu, S., Yang, R., Huang, S., Xu, J., & Wang, Z. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00076 · Cited by 20
Li, F., Zhao, S., Qing, Y., Chen, X., Guan, X., Wang, S., Zhang, G., & Cui, H. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3247883 · Cited by 20
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter PretrainingPreprint
Lin, J., An, Y., Bai, J., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Li, Y., Lin, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03888 · Cited by 23
Performance Characterization of CXL Memory and Its Use Cases
Wang, X., Liu, J., Wu, J., Yang, S., Ren, J., Shankar, B., & Li, D. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00097 · Cited by 16
Self-citationProteus: Simulating the Performance of Distributed DNN Training
Duan, J., Li, X., Xu, P., Zhang, X., Yan, S., Liang, Y. E., & Lin, D. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3443255 · Cited by 23
Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly
Woisetschläger, H., Erben, A., Wang, S., Mayer, R., & Jacobsen, H. (2024) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3650203.3663331 · Cited by 32
RAMP: A flat nanosecond optical network and MPI operations for distributed deep learning systems
Ottino, A., Benjamin, J., & Zervas, G. (2023) · Optical Switching and Networking · doi:10.1016/j.osn.2023.100761 · Cited by 14
FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation
Kim, T., Park, C., Mukimbekov, M., Hong, H., Kim, M., Jin, Z., Kim, C., Shin, J.-Y., & Jeon, M. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636238 · Cited by 14
MAGIS: Memory Optimization via Coordinated Graph Transformation and Scheduling for DNN
Chen, R., Ding, Z., Zheng, S., Zhang, C., Leng, J., Liu, X., & Liang, Y. E. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651330 · Cited by 16
Han, H., Zheng, X., Wen, Y., Hao, Y., Feng, E., Liang, L., Mu, J., Li, X., Ma, T., Jin, P., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674168 · Cited by 13
AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning
Singh, S., & Bhatele, A. S. (2022) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps53621.2022.00065 · Cited by 16
M6-T: Exploring Sparse Expert Models and BeyondPreprint
An, Y., Lin, J., Men, R., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Wang, J., Li, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15082 · Cited by 60
A Cost-Efficient Failure-Tolerant Scheme for Distributed DNN Training
Chen, M., Hua, Y., Bai, R., & Huang, J. (2023) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd58817.2023.00031 · Cited by 13
Enabling Large Dynamic Neural Network Training with Learning-based Memory Management
Ren, J., Xu, D., Yang, S., Zhao, J., Li, Z., Navasca, C., Wang, C., Xu, H., & Li, D. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00066 · Cited by 14
Self-citationMaurya, A., Ye, J., Rafique, M. M., Cappello, F., & Nicolae, B. (2024) · doi:10.1145/3659995.3660038 · Cited by 11
PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption
Tan, Y., Tan, C. L., Mi, Z., & Chen, H. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707224 · Cited by 18
Distributed training of large language models: A survey
Zeng, F., Gan, W., Wang, Y., & Yu, P. S. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100174 · Cited by 15
Momentum-driven adaptive synchronization model for distributed DNN training on HPC clusters
Zhang, Z., Ji, Z., & Wang, C. (2021) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2021.09.007 · Cited by 11
CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs
Sun, Q., Liu, Y., Yang, H., Zhang, R., Dun, M., Li, M., Liu, X., Xiao, W., Li, Y., Luan, Z., et al. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00044 · Cited by 11
InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models
Nagrecha, K., Liu, L., Delgado, P., & Padmanabhan, P. (2023) · ACM Conference on Recommender Systems · doi:10.1145/3604915.3608778 · Cited by 10
Deep Optimizer States: Towards Scalable Training of Transformer Models using Interleaved Offloading
Maurya, A., Ye, J., Rafique, M. M., Cappello, F., & Nicolae, B. (2024) · International Middleware Conference · doi:10.1145/3652892.3700781 · Cited by 10
Buffalo: Enabling Large-Scale GNN Training via Memory-Efficient Bucketization
Yang, S., Zhang, M., & Li, D. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00083 · Cited by 10
Self-citationMixPipe: Efficient Bidirectional Pipeline Parallelism for Training Large-Scale Models
Zhang, W., Zhou, B., Tang, X., Zhaoxing, W., & Hu, S. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247730 · Cited by 12
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed TrainingPreprint
Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I. G., Mahoney, M. W., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14129 · Cited by 13
Breaking Barriers: Expanding GPU Memory with Sub-Two Digit Nanosecond Latency CXL Controller
Gouk, D., Kang, S.-K., Bae, H., Ryu, E., Lee, S., Kim, D., Jang, J., & Jung, M. (2024) · USENIX Workshop on Hot Topics in Storage and File Systems · doi:10.1145/3655038.3665953 · Cited by 12
Implementing CUDA Unified Memory in the PyTorch Framework
Choi, J., Yeom, H. Y., & Kim, Y. (2021) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C) · doi:10.1109/acsos-c52956.2021.00029 · Cited by 9
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model TrainingPreprint
Karakus, C., Huilgol, R., Wu, F. Q., Subramanian, A., Daniel, C., Çavdar, D., Xu, T., Chen, H., Rahnama, A., & Quintela, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05972 · Cited by 29
Sparse Fusion for Multimodal TransformersPreprint
Ding, Y., Rich, A., Wang, M. L., Stier, N., Turk, M., Sen, P., & Höllerer, T. (2022) · doi:10.31219/osf.io/f7jvn · Cited by 9
A Survey on Performance Modeling and Prediction for Distributed DNN Training
Guo, Z., Tang, Y., Zhai, J., Yuan, T., Jin, J., Li, W., Zhao, Y., & Li, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3476390 · Cited by 12
Machine Learning-Guided Memory Optimization for DLRM Inference on Tiered Memory
Ren, J., Ma, B., Yang, S., Francis, B., Ardestani, E. K., Si, M., & Dong, L. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00121 · Cited by 11
Self-citationGKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Tan, S., Tam, W. L., Wang, Y., Gong, W., Zhao, S., Zhang, P., & Tang, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.15 · Cited by 17
Adaptive Sparse Deep Neural Network Inference on Resource-Constrained Cost-Efficient GPUs
Dun, M., Zhang, X., Cao, H., Zhang, Y., Huang, J., & Ye, X. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363545 · Cited by 8
Watching a Language Model Learning Chess
Stöckl, A. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_153 · Cited by 8
BMCook: A Task-agnostic Compression Toolkit for Big Models
Zhang, Z., Gong, B., Chen, Y., Han, X., Zeng, G., Zhao, W., Chen, Y., Liu, Z., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.40 · Cited by 8
Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Lin, Z., Miao, Y., Xu, G., Li, C., Saarikivi, O., Maleki, S., & Yang, F. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00067 · Cited by 11
Efficient inference, training, and fine-tuning of protein language models
Çelik, M. H., & Xie, X. (2025) · iScience · doi:10.1016/j.isci.2025.113495 · Cited by 7
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Jiang, Y., Fu, F., Miao, X., Nie, X., & Cui, B. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/238 · Cited by 7
Bandwidth Characterization of DeepSpeed on Distributed Large Language Model Training
Hanindhito, B., Patel, B., & John, L. K. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00031 · Cited by 11
APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes
Wei, Y., Du, J., Jiang, J., Shi, X., Zhang, X., Huang, D., Xiao, N., & Lu, Y. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00096 · Cited by 9
pommDNN: Performance optimal GPU memory management for deep neural network training
Chen, W., Dong, X., Chen, X. F., Liu, S., Qin, X., & Wang, Q. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.10.025 · Cited by 7
RT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference
Kang, W., Lee, J., Lee, Y., Oh, S., Lee, K., & Chwa, H. S. (2024) · IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) · doi:10.1109/rtas61025.2024.00037 · Cited by 6
Survey on Efficient Training of Large Neural Networks
Gusak, J., Cherniuk, D., Shilova, A., Katrutsa, A., Bershatsky, D., Zhao, X., Eyraud‐Dubois, L., Shliazhko, O., Dimitrov, D., Oseledets, I. V., et al. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/769 · Cited by 6
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Anthony, Q. G., Awan, A. A., Rasley, J., He, Y., Shafi, A., Abduljabbar, M., Subramoni, H., & Panda, D. K. D. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00103 · Cited by 6
Self-citationA Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 13
Kanakarajan, K. R., & Sankarasubbu, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.137 · Cited by 7
SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers
Xu, D., Xu, M., Lou, C., Zhang, L. L., Huang, G., Jin, X., & Liu, X. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3617232.3624847 · Cited by 6
Scalable and Load-Balanced Full-Graph GNN Training on Multiple GPUs
Wang, Q., Chen, Y., Wong, W., & He, B. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3558641 · Cited by 5
Long-Term Ad Memorability: Understanding & Generating Memorable Ads
Harini, S. I., Singh, S., Singla, Y. K., Bhattacharyya, A., Baths, V., Chen, C., Shah, R. R., & Krishnamurthy, B. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00557 · Cited by 16
Metadata
- Authors
- Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y.
- Affiliations
- University of California, Merced (US)
- Microsoft (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2101.06840
- OpenAlex
- W3121562065
- Cited by
- 151
- References
- 27
- Citations by year
- 2025: 20 · 2024: 46 · 2023: 44 · 2022: 16 · 2021: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- large-scale model training, GPU memory efficiency, CPU offloading, data movement reduction, single-GPU training, model parallelism, distributed training, PyTorch
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
