The DOI matches its Crossref record; 37 of the 39 references it cites were found in a registry or an index.
Efficient large-scale language model training on GPU clusters using megatron-LM
Open accessAbstract
Large language models have led to state-of-the-art accuracies across several tasks. However, training these models efficiently is challenging because: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to scaling issues at thousands of GPUs. In this paper, we show how tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs. We propose a novel interleaved pipelining schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput of 52% of theoretical peak).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages.”
From The Llama 3 Herd of Models (2024) · cites this paper as Narayanan et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362
Self-citation
Research works printed but not linked: 3
- Paras Jain , Ajay Jain , Aniruddha Nrusimha , Amir Gholami , Pieter Abbeel , Joseph Gonzalez , Kurt Keutzer , and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020 , pages 497 -- 511 . 2020 . Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pages 497--511. 2020.
- Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving Language Understanding by Generative Pre-Training , 2018 . Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners . OpenAI Blog , 1 ( 8 ): 9 , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019.
Cited by
Sorted by citations
Items 601–650 of 650
PipeComm: Maximizing Link Utilization Through Pipeline-Aware Collective Communication Synthesis
Xu, R., Luo, Y., Meng, Y., Zheng, S., Li, M., & Liang, Y. E. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00152
BCD-Megatron: A Cost-Effective Training System for Large Language Models
Liu, Z., Li, Y., Zhang, Y., Zhang, B., Jiang, G., Zhang, X., Xiao, L., Zhang, W., & Cheng, D. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00068
Pulse: Training Acceleration for Large Diffusion Models with Automatic Pipeline Parallelism
Sun, B., Jiang, G., Zhang, L., Chen, C., Tao, Y., Che, Z., Yu, J., Chang, S., Gu, H., Liu, F., et al. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00041 · Cited by 1
Optimizing Wide-Area Collective Communication via Flow-Dependency-Aware Rate Allocation
Wang, Y., Craps, T., & Li, B. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00131
Mapping and Communication Optimizations with Fault Tolerance for Wafer-Scale LLM Inference
Cui, J., Qin, L., Cai, W., & Huang, J. (2026) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca66397.2026.00076
Tensor Parallelism for State-Space Models
Dutt, A., Shah, N., Masarani, H., & Gandhi, A. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00074
Aligning Large Language Models with Searcher Preferences
Wu, W., Zhou, P., Chen, L., Wang, Q., Lu, C., Gao, Y. F., Wu, Y. H., Hu, Y., & Xiong, H. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818318
One-chip-like datacenter design enabled by CXL-based scale-up fabrics
Jung, M., Woo, H., Kim, J., Baek, J., Nam, K., Na, E., Choi, H., 장성현, Choi, H., Shakeri, K., et al. (2026) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-026-00315-5
Theseus: Runtime-Adaptive GPU Collective Communication with Hot-Swappable Schedules
Ding, R., Lu, X., Wang, J., Liu, X., Wang, F., Hao, X., Zhu, H., Xu, A., Cao, S., Sun, H., et al. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829134
Spillway: Orchestrating DPU and Host into a Unified vSwitching Fabric
Jiang, X., Fan, D., Lv, Y., Guan, N., Zhao, Q., Fu, S., Yang, Y., Zhou, Y., Ge, X., Wu, D., et al. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829154
Harvest: Adaptive Photonic Switching Schedules for Collective Communication in Scale-up Domains
Rahman, M., Joseph, S., Kodkani, N., Arzani, B., & Addanki, V. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829166
Li, G., Li, Y., Zang, M., & Wu, J. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829174
MonkeyTree: Near-Minimal Congestion for Multi-tenant Training via Migration
Zabreyko, A. A., Wang, W., & Ghobadi, M. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829150
STORM: Enabling Traffic Scheduling for RDMA
Wu, J., Shu, R., Antichi, G., Xiong, Y., Moore, A. W., & Crowcroft, J. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829117
Turbo: Efficiently Serving Long-Context Large Language Models with In-Network Aggregation
Wan, Y., Xu, Y., Zhang, C., Huang, Y., Feng, Y., Xu, W. L., Li, J., Xu, M., Wu, W., & Miao, C. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829140
DLRover-LM: LLM Pre-Training Framework With Thousands of Accelerators in AntGroup
Huang, Z.-L., Ye, Z.-M., Cai, Q.-S., Huang, Z.-L., Sang, B., Zhang, H., Sha, J., Lan, T., Lu, H., Zhou, Y.-C., et al. (2026) · IEEE International Conference on Data Engineering · doi:10.1109/icde65706.2026.00271
Tong, Q., Hong, W., Chen, M., Chen, L., Zheng, J., Zhou, C., An, R., & Jia, L. (2026) · doi:10.1109/iccms69991.2026.11642775
Yang, J., Liu, T., Pei, J., Wu, Q., & Guan, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35251-4_18
A rapid random flow generator based on ‘GPU stream cubes’ for turbulent wind fields
Liao, S., Huang, M., Guo, C., Li, Q., Cai, K., & Wang, L. (2026) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2026.2711525
Hierarchical One-Link Interconnection Networks for Low-Degree Parallel Communication
Soe, H. N., Zhang, Y., & Xu, Z. (2026) · Parallel Processing Letters · doi:10.1142/s0129626426500155
Conductor: Dynamically orchestrating pipeline parallelism with multi-granularity control
Dong, X., Liu, Z., Yang, Y., Lai, Y.-L., & Jin, Z. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108762
Memory-Safe Hierarchical Planning for Hybrid-Parallel Transformer Training
Zhang, Z., Zhang, Z., Wang, N., Wang, J., & Zhao, J. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3842760
DOA: Dataflow Optimization for Attention on Multi-core DSPs with Three-Level Memory Hierarchy
Li, S., Lai, Z., Xu, S., Ouyang, S., Xiao, H., Zhang, Z., Jia, M., Su, H., & Li, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_25
Yuan, L., Qiao, L., Tang, Y., Zhu, M., Shang, W., & Li, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-4805-6_1
TAILOR: Token-Aware Partitioning and Routing for Edge–Cloud Transformer Inference
Huang, X., Liu, R. R., & Wu, J. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661040
ATAC: Anchor-tail aware context parallelism for LLM training
Liu, Z., Hou, S., Gao, Y., Qi, X., Zhang, X., & Zhang, F. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00910-4
PRISM: Workload-Aware Autonomous Network Fault Recovery for Hyperscale AI Training Fabrics
Bhosle, R. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00045
DTCO study for in-package memory scaling with Active Interposers
Patel, A., Akhunov, K., Pantano, N., Ryckaert, J., Tökei, Z., Kar, G. S., & Biswas, D. (2026) · IEEE International Conference on IC Design and Technology (ICICDT) · doi:10.1109/icicdt69693.2026.11669168
On‐Chip Learning With Crossbar Arrays for Adaptive Edge Intelligence
James, A. P. (2026) · Advanced Computing · doi:10.1002/adco.70006
Chameleon: Adaptive Bidirectional Pipeline Parallelism for Edge Collaborative Training
Liu, T., Hou, X., Li, L., Cong, P., Meng, D., & Zhou, J. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832895
Memory-Aware Joint Optimisation of Partitioning and Scheduling for Pipeline-Parallel Training
Wang, N., Raith, A., & Sinnen, O. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832891
Li, Y., Pang, T., Cai, Z., Pan, Z., Mao, Y., & Wu, J. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832886
OmniPipe: Efficient, Flexible and Scalable Pipeline Parallelism for Large Model Training
Li, J., Ma, Z., & Li, S. (2026) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3832810.3832820
RecDM: Efficient Training System for Large-Scale Recommendation Models on Disaggregated Memory
Wang, Z., Yu, Z., Wang, K., Lin, Y., Li, Y., Liu, L., Tang, X., Wang, Y., Kang, Y., & Ding, Y. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3848634
Li, H., Lin, S., Fu, F., Zhou, Y., Ji, X., Zhao, Y., Wang, L., Jiang, J., & Cui, B. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3837125 · Cited by 15
MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference
He, C., Xu, L., Lu, Z., Huang, Y., Xiao, H., Deng, C., Ma, L., Miao, Z., Yang, F., & Mai, L. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843856
Taming Dynamism on GPUs: Cross-SM Kernel Fusion via SM Cooperation and Just-in-Time Reduction
Jingkai, H., Sun, G., Li, T., Du, D., Xia, Y., & Chen, H. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843857
Janus: Multi-LLM Serving at Production Scale
Zhou, T., Wang, Y., Zhou, Y., Liu, Z., Wang, Z., Peng, Y., Wang, Y., Zhang, Y., Yin, J., Tian, K., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843867
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with CrystalLLM
Xi, S., Lao, C., Jia, B., Gao, J., Zhang, Z., Cao, J., Sutioso, B., Xu, E., Yu, M., Ren, K., et al. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843852
Nelapatla, P. R., Starkey, E., & Zhou, Y. X. (2026) · Electronics · doi:10.3390/electronics15194473
Photonic neuromorphic learning via generalized in situ physical gradient descent
Zhou, T., Zhao, Y., Li, S., Shao, G., Huang, R., & Fang, L. (2026) · Nature Computational Science · doi:10.1038/s43588-026-01057-y
Telemetry-Aware Load Balancing Method and Algorithm for Distributed LLM Training on AI Servers
Popov, A. V., Potapov, A. V., & Kugoev, I. A. (2026) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon69876.2026.11707335
Large language models linking traditional Chinese medicine knowledge and clinical practice
Li, C., Zhang, X., Zhu, L., Zhu, B., Chu, C., Li, X., Yao, Y., Zeng, Y., & Zheng, J. (2026) · Chinese Medicine · doi:10.1186/s13020-026-01489-8
Raj, S., Sai, S., Simmhan, Y. L., Chard, K., & Buyya, R. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108858 · Cited by 3
Diagnosing Pipeline Inefficiencies in Large-Scale LLM Training on GPU Clusters
Chen, J., Xuan, Z., Zhang, Y., Wang, X., You, X., & Yang, H. (2026) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster68337.2026.00019
Artemis: Accelerating Mixture-Of-Experts Training Via Dynamic Expert Parallelism
Meng, Z., Diao, L., Qian, Z., Lin, W., & Wu, C. (2026) · IEEE International Conference on Cluster Computing (CLUSTER) · doi:10.1109/cluster68337.2026.00018
Scalable Training Architecture for Next Generation Language Models
Sangam, G. K., Bhargav, N., Dandigam, R., BasiReddy, S. R., Pandey, D., & Sadat, Q. T. (2026) · doi:10.1109/incect72370.2026.11742771
Heterogeneous GPU Training as a Contract: A Six-Enabler Framework for Cross-Architecture Pipelines
Okumuş, F., & Altuntaş, Y. (2026) · 2026 10th International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728232
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771 · Cited by 7
“However, their immense scale, reaching hundreds of billions or trillions of parameters [2, 3], leads to prohibitive computational and memory costs, particularly GPU VRAM, hindering their widespread deployment on consumer or edge devices [4–6].”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Metadata
- Authors
- Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M.
- Affiliations
- NVIDIA and Microsoft Research
- Nvidia (United Kingdom) (GB)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
- Publisher
- ACM
- Published
- 2021-11-13
- Pages
- 1-15
- ISSN
- 2167-4337 (ISSN-L)
- OpenAlex
- W3187255235
- Cited by
- 639
- References
- 39
- Citations by year
- 2026: 151 · 2025: 211 · 2024: 152 · 2023: 90 · 2022: 33 · 2021: 2
- FWCI
- 50.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-165651 8, 1651570, 1656518, CAREER)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model training, GPU clusters, tensor parallelism, pipeline parallelism, data parallelism, memory efficiency, training throughput
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
