PreprintCited by 153Parallel Computing and Optimization TechniquesAdvanced Neural Network ApplicationsStochastic Gradient Optimization Techniques

The DOI matches its DataCite record; 26 of the 27 references it cites were found in a registry or an index.

ZeRO-Offload: Democratizing Billion-Scale Model Training

Open access

Ren, Jie, Rajbhandari, Samyam, Aminabadi, Reza Yazdani, Ruwase, Olatunji, Yang, Shuangyan, Zhang, Minjia, Li, Dong, He, Yuxiong·2021·arXiv

DOI 10.48550/arxiv.2101.06840arXiv 2101.06840

Open this paper in CoChat — free account Open PDF

Abstract

Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2101.06840

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 3×
“Recent studies in fact show that larger models are more resource-efficient to train than smaller ones [12] for a given accuracy target.”
BackgroundMethodInfluential
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
PyTorch distributed
Shen, L., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J. S., Vaughan, B., Damania, P., et al. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415530 · Cited by 996

Show all 27 references

Research works printed but not linked: 7

[17][17] Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS '20, page 891-905, New York, NY, USA, 2020. Association for Computing Machinery. -
[19][19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
[21][21] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , 2020. -
[26][26] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism on Neural Network Training. Journal of Machine Learning Research , 20, 2019. -
[27][27] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Meshtensorflow: Deep learning for supercomputers. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 10435-10444, 2018. -
[30][30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. -
[33][33] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , PPoPP '18, page 41-53, New York, NY, USA, 2018. Association for Computing Machinery. -

Sorted by citations

Items 101–153 of 153

MSCH: Microbatch-Based Selective Activation Checkpointing With Recomputation Hidden for Efficient Training of LLM Models
Zhang, C., Yu, M., Yu, L., Cong, P., Yuchao, Y., Bao, J., Jian, J., Wang, X., Ye, X., Tang, T., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3456788 · Cited by 6
How Much Storage Do We Need for High Performance Server
Wei, J., & Zhang, X. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00303 · Cited by 5
Tensor Movement Orchestration in Multi-GPU Training Systems
Lin, S., Chen, Y.-J., Cheng, H.-Y., & Yang, C.-L. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071043 · Cited by 6
Pretraining Billion-Scale Geospatial Foundational Models on Frontier
Tsaris, A., Dias, P. A., Potnis, A. V., Yin, J., Wang, F., & Lunga, D. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00174 · Cited by 11
Innovative Adapter-Based Fine-Tuning and Streamlined Training Strategies for Text Summarization
Damana, A. A., & Hitesh, P. (2024) · International Conference on Electrical and Computer Engineering Researches (ICECER) · doi:10.1109/icecer62944.2024.10920340 · Cited by 4
Ratel: Optimizing Holistic Data Movement to Fine-tune 100B Model on a Consumer GPU
Liao, C., Sun, M., Yang, Z., Xie, J., Chen, K., Yuan, B., Wu, F., & Wang, Z. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00029 · Cited by 27
Performance Analysis and Optimization of Nvidia H100 Confidential Computing for AI Workloads
Tan, Y., & Mi, Z. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00192 · Cited by 5
Whale: A Unified Distributed Training Framework.Preprint
Wang, A., Jia, X., Jiang, L., Zhang, J., Li, Y., & Lin, W. (2020) · arXiv (Cornell University) · Cited by 3
Long-term series forecasting with Query Selector -- efficient model of sparse attentionPreprint
Klimek, J., Klimek, J., Kraśkiewicz, W., & Topolewski, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08687 · Cited by 4
Hybrid Parallel Inference for Large Model on Heterogeneous Clusters for High Throughput
Xu, C., Kong, W., Liu, M., Zhang, M., Li, C., & Gong, L. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00087 · Cited by 3
Self-citation
ZeROf-Offload: forward-gradient scheme for efficient full parameter fine-tuning of billion-scale language models
Zhu, J. F., Feng, P., Lu, J., Fang, B., & Yang, H. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad9667 · Cited by 5
Adam Accumulation to Reduce Memory Footprints of Both Activations and Gradients for Large-Scale DNN Training
Zhang, Y., Han, Y., Cao, S., Dai, G., Miao, Y., Cao, T., Yang, F., & Xu, N. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230623 · Cited by 9
COSMOS: Coordinated Management of Cores, Memory, and Compressed Memory Swap for QoS-Aware and Efficient Workload Consolidation for Memory-Intensive Applications
Han, M., Park, E., Shin, Y., Oh, D.-J., Cho, Y., & Baek, W. (2023) · IEEE Access · doi:10.1109/access.2023.3336685 · Cited by 3
Automated Insights Into GitHub Collaboration Dynamics
Bian, J., Arefyev, N. V., Mühlhäuser, M., & Welzl, M. (2025) · IEEE Access · doi:10.1109/access.2025.3566309 · Cited by 3
Prompting for explanations improves Adversarial NLI. Is this true? {Yes} it is {true} because {it weakens superficial cues}
Kavumba, P., Brassard, A., Heinzerling, B., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.162 · Cited by 4
XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments
Schuler, M., Membarth, R., & Slusallek, P. (2022) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3568956 · Cited by 3
A Survey of Backpropagation-free Training For LLMSPreprint
Mei, H., Cai, D., Wu, Y., Wang, S., & Xu, M. (2024) · doi:10.36227/techrxiv.171172909.97532161/v1 · Cited by 3
Optimization of tokenization and memory management for processing large textual corpora in multilingual applications
Dodić, D., Regodić, D., & Milutinović, N. (2025) · Bulletin of Natural Sciences Research · doi:10.5937/bnsr15-54993 · Cited by 3
Overlapped Data Processing Scheme for Accelerating Training and Validation in Machine Learning
Choi, J., & Kang, D. H. (2022) · IEEE Access · doi:10.1109/access.2022.3189373 · Cited by 3
Parallel intelligent computing: development and challenges
卢, 凯., 赖, 志., 李, 笙., 柳, 炜., 葛, 可., 卢, 锡., & 李, 东. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0051 · Cited by 2
Leveraging Memory Expansion to Accelerate Large-Scale DL Training
Kadiyala, D. K., Rashidi, S., Heo, T., Bambhaniya, A. R., Krishna, T., & Daglis, A. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00035 · Cited by 2
Sheared Backpropagation for Fine-Tuning Foundation Models
Yu, Z., Shen, L., Ding, L., Tian, X., Chen, Y., & Tao, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00562 · Cited by 3
Latent Weight Quantization for Integerized Training of Deep Neural Networks
Fei, W., Dai, W., Zhang, L., Zhang, L., Li, C., Zou, J., & Xiong, H. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3527498 · Cited by 4
Pinpointing the Memory Behaviors of DNN Training
Li, J., Dong, X., Li, G., Zhao, P., Wang, X., Chen, X., Yu, X., Yang, Y., Jiang, Z., Cao, W., et al. (2021) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass51385.2021.00038 · Cited by 2
Whale: Scaling Deep Learning Model Training to the Trillions.Preprint
Jia, X., Jiang, L., Wang, A., Zhang, J., Li, X., Xiao, W., Chen, L., Li, Y., Zheng, Z., Liu, X., et al. (2020) · arXiv (Cornell University) · Cited by 2
HSDP: Accelerating Large-scale Model Training via Efficient Sharded Data Parallelism
Hao, Y., Lai, Z., Wang, W., Li, S., Liu, W., Ge, K., & Li, D. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00023 · Cited by 1
Self-citation
Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?
Section of Communication Systems Bachelor semester 6 EPFL, L., Blin, K., & Kucharavy, A. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_021 · Cited by 1
ScaMP: Scalable Meta-Parallelism for Deep Learning Search
Anthony, Q. G., Xu, L., Shafi, A., Subramoni, H., & Panda, D. K. D. (2023) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid57682.2023.00044 · Cited by 1
Security of NVMe Offloaded Data in Large-Scale Machine Learning
Krauß, T., Götz, R., & Dmitrienko, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-51482-1_8 · Cited by 1
Energy-Efficient Deep Learning Training
Guan, L., Zhang, S., & Chen, Y. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1007491 · Cited by 1
Automatic Parallelization with CodeT5+: A Model for Generating OpenMP Directives
Pornmaneerattanatri, S., Takahashi, K., Kashiwa, Y., Ichikawa, K., & Iida, H. (2024) · IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops) · doi:10.1109/clusterworkshops61563.2024.00028 · Cited by 1
ParaCkpt: Heterogeneous Multi-Path Checkpointing Mechanism for Training Deep Learning Models
Wang, S., Cao, Q., Zhou, K., Xu, J., Guo, Z., & Guo, J. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00036 · Cited by 1
Towards robust and generalizable representations of extracellular data using contrastive learningPreprint
Vishnubhotla, A., Loh, C., Paninski, L., Srivastava, A., & Hurwitz, C. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.30.564831 · Cited by 1
JUWELS Booster -- A Supercomputer for Large-Scale AI ResearchPreprint
Kesselheim, S., Herten, A., Krajsek, K., Ebert, J., Jitsev, J., Cherti, M., Langguth, M., Gong, B., Stadtler, S., Mozaffari, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.11976 · Cited by 3
Toward efficient resource utilization of a GPU-accelerated AI supercomputer
Arora, R. (2021) · doi:10.17760/d20412322
ScaMP: Scalable Meta-Parallelism for Deep Learning Search
Anthony, Q. G., Xu, L., Shafi, A., Subramoni, H., & Panda, D. K. D. (2023) · doi:10.1109/ccgridw59191.2023.00080
Optimizing Single DGX-A100 System: Overcoming GPU Limitations via Efficient Parallelism and Scheduling for Large Language Models
Kim, K.-H., & Jeong, C. (2023) · Applied Sciences · doi:10.3390/app13169306
Save GPU RAM Usage in Convolutional Layers to Load Larger Histopathological ImagesPreprint
Ando, T. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.19.558533
Multi-level Storage Optimization for Intermediate Data in AI Model Training
Fu, J., Yang, Y., Hu, G., Luo, X., & Shao, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47843-7_11
Remember what you did so you know what to do next
Ciosici, M. R., Hedges, A., Kankanampati, Y., Martin, J. D., Freedman, M., & Weischedel, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.104
Efficient Large Models Fine-tuning on Commodity Servers via Memory-balanced Pipeline Parallelism
Liu, Y., Lai, Z., Liu, W., Wang, W., & Li, D. (2023) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys60770.2023.00103
Self-citation
NDPipe: Exploiting Near-data Processing for Scalable Inference and Continuous Training in Photo Storage
Kim, J., Oh, S., Kung, J., Kim, Y., & Lee, S. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651345 · Cited by 3
Structural Segmentation and Large Language Model Make Behavior Tree More Explainable
Su, H., Zhao, X., Li, F., Guo, Z., Wu, Y., & Wang, Y. (2023) · International Conference on Artificial Intelligence, Human-Computer Interaction and Robotics (AIHCIR) · doi:10.1109/aihcir61661.2023.00056
Mbapp: Efficient Memory-Balanced Pipeline Parallelism for Large Model Fine-Tuning on Commodity GPU Servers
Liu, Y., Lai, Z., & Li, D. (2024) · International Conference on Computer Information and Big Data Applications (CIBDA) · doi:10.1145/3671151.3671153
Self-citation
Reinforcement Learning for Intra- & Inter-Node Recommender Data Pipeline Optimization
Nagrecha, K., Liu, L.-Y., & Delgado, P. (2024) · Trans. Recomm. Syst. · doi:10.1145/3704923
Efficient Data-parallel Distributed DNN Training for Big Dataset under Heterogeneous GPU Cluster
Ahn, S., Lee, S., Choi, H., & Lee, J. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825722
Towards ML System Extensibility
Deng, W., Ruan, A., Frisella, M., Chen, K.-H., Cho, S., Humphries, J. T., Qiao, R., & Wang, S. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730392
DOME: Dynamic Optimization of GPU Memory for Training DNN Models with Limited Resources
Ma, Z., Jian, S., & Tan, Y. (2024) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn63567.2024.00102
Memory Efficient LM Compression Using Fisher Information from Low-Rank Representations
Moskovskiy, D., Pletenev, S., Zagoruyko, S., & Panchenko, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97141-9_2
Towards Robust and Scalable Mixture of Experts Architectures for Large Language and Vision ModelsPreprint
Yousra, A., Fawziya, J., & Gamal, F. (2025) · doi:10.31224/4764
Scalable and Interpretable Mixture of Experts Models in Machine Learning: Foundations, Applications, and ChallengesPreprint
Jafar, R., Gamal, F., & Raheem, R. (2025) · Preprints.org · doi:10.20944/preprints202507.0283.v1
Scalability And Resilience In Distributed LLM Training: A Survey Of Modern Techniques
Gupta, S. (2025) · Journal of International Crisis and Risk Communication Research · doi:10.63278/jicrcr.vi.3189
The Llama 3 Herd of Models
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citation

Authors: Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y.
Affiliations: University of California, Merced (US)
Microsoft (United States) (US)
ORCID: Ren, Jie
Rajbhandari, Samyam
Aminabadi, Reza Yazdani
Ruwase, Olatunji
Zhang, Minjia
Li, Dong
He, Yuxiong
Venue: arXiv
Publisher: arXiv
Published: 2021
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2101.06840
arXiv: 2101.06840
OpenAlex: W3121562065
Cited by: 151
References: 27
Citations by year: 2025: 20 · 2024: 46 · 2023: 44 · 2022: 16 · 2021: 23
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: creativecommons.org/licenses/by/4.0/legalcode
Locations: arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University) (cc-by)
Topics: Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords: large-scale model training, GPU memory efficiency, CPU offloading, data movement reduction, single-GPU training, model parallelism, distributed training, PyTorch

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ZeRO-Offload: Democratizing Billion-Scale Model Training

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks