CoChat
Sign Up
PreprintCited by 893Topic ModelingParallel Computing and Optimization TechniquesNatural Language Processing Techniques

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Open access
Shoeybi, Mohammad, Patwary, Mostofa, Puri, Raul, LeGresley, Patrick, Casper, Jared, Catanzaro, Bryan·2019·arXiv
DOI 10.48550/arxiv.1909.08053arXiv 1909.08053
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.08053

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • FlashAttention achieves 3x speedup on GPT-2 (seq length 1K) compared to baselines.

    “3 speedup on GPT-2 (seq. length 1K)”

    From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [77]

    Appendix D.2 (p. 14) reports a 2.98-fold training speedup for a 1.2-billion-parameter model using eight GPUs rather than one. That result concerns model parallelism, not FlashAttention; the passage does not specify a 1K sequence length or the claimed baseline comparison. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Training efficiency can be improved by overlapping computation and communication in specific scenarios, such as GEMM and all-to-all sequence in expert parallelism, or GEMM followed by AllReduce and ReduceScatter in tensor parallelism.

    “Furthermore, training efficiency can be improved by overlapping computation and communication in specific scenarios, such as general matrix multiplication (GEMM) and all-to-all sequence in expert parallelism [159], or GEMM followed by AllReduce and ReduceScatter in tensor parallelism [149].”

    From A Survey of Large Language Models (2026) · cites this paper as [149]

    Section 3 (p. 4) describes tensor-parallel GEMMs followed by an all-reduce and says that eliminating an intermediate synchronization point improves scaling. It does not describe overlapping computation with communication or using ReduceScatter in that sequence. The expert-parallelism example is attached to a different citation [159], so it is not assessed here. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Within a machine, high NVLink bandwidth enables the use of tensor parallelism.

    “Within each machine, the high NVLink bandwidth enables the use of tensor parallelism (Shoeybi et al., 2019).”

    From The Llama 3 Herd of Models (2024) · cites this paper as Shoeybi et al., 2019

    The Experiments section (p. 6) reports high bandwidth between GPUs within a server, and Appendix B.1 (p. 11) describes model-parallel GPU groups within a server. But the paper identifies the intra-server interconnect as NVSwitch, not NVLink, and does not explicitly say that its bandwidth enables tensor parallelism. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • GPT-2 small and medium using FlashAttention achieve up to 3x speedup compared to Huggingface and up to 1.7x compared to Megatron-LM.

    “Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”

    From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [77]

  • Various forms of parallelism have been developed to scale the training of LLMs across hardware clusters, each targeting different computational or memory bottlenecks.

    “To scale the training of LLMs across hardware clusters, various forms of parallelism have been developed, each targeting different computational or memory bottlenecks [149,150].”

    From A Survey of Large Language Models (2026) · cites this paper as [149]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layernorm. CoRR , abs/1607.06450, 2016. URL http://arxiv . org/ abs/1607 . 06450 . · arXiv
  2. Chen, C.-C., Yang, C.-L., and Cheng, H.-Y. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv:1809.02839 , 2018. · arXiv
  3. Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. CoRR , abs/1604.06174, 2016. URL http://arxiv . org/ abs/1604 . 06174 . · arXiv
  4. Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR , abs/1901.02860, 2019. URL http://arxiv . org/ abs/1901 . 02860 . · arXiv
  5. He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR , abs/1706.02677, 2017. · arXiv
  6. Hendrycks, D. and Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR , abs/1606.08415, 2016. URL http: //arxiv . org/abs/1606 . 08415 . · arXiv
  7. Howard, J. and Ruder, S. Fine-tuned language models for text classification. CoRR , abs/1801.06146, 2018. · arXiv
  8. Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, Q. V., and Chen, Z. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR , abs/1811.06965, 2018. URL http://arxiv . org/ abs/1811 . 06965 . · arXiv
  9. Jia, Z., Zaharia, M., and Aiken, A. Beyond data and model parallelism for deep neural networks. arXiv:1807.05358 , 2018. · arXiv
  10. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large- batch training for deep learning: Generalization gap and sharp minima. ICLR , 2017.
  11. Lan, Z., Chen, M., Goodman, S., Gimpel, K., and Soricut, P. S. R. Albert: A lite bert for self-supervised learning of language representations. arXiv:1909.11942 , 2019. · arXiv
  12. Liu, X., He, P., Chen, W., and Gao, J. Multi-task deep neural networks for natural language understanding. CoRR , abs/1901.11504, 2019a. URL http://arxiv . org/ abs/1901 . 11504 . · arXiv
  13. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. URL https:// openreview . net/forum?id=Bkg6RiCqY7 .
  14. McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. CoRR , abs/1708.00107, 2017. · arXiv
  15. Melamud, O., Goldberger, J., and Dagan, I. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning , pp. 51-61, 01 2016.
  16. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. CoRR , abs/1609.07843, 2016. URL http://arxiv . org/abs/1609 . 07843 . · arXiv
  17. Micikevicius, P., Narang, S., Alben, J., Diamos, G. F., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. CoRR , abs/1710.03740, 2017. · arXiv
  18. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and ˇ Cernock` y, J. Empirical evaluation and combination of advanced language modeling techniques. In Twelfth Annual Conference of the International Speech Communication Association , 2011.
  19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed representations of words and phrases and their compositionality. CoRR , abs/1310.4546, 2013. · arXiv
  20. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683 , 2019. · arXiv
  21. Rajpurkar, P., Jia, R., and Liang, P. Know what you dont know: Unanswerable questions for squad. ACL , 2018.
  22. Ramachandran, P., Liu, P. J., and Le, Q. V. Unsupervised pretraining for sequence to sequence learning. CoRR , abs/1611.02683, 2016. URL http://arxiv . org/ abs/1611 . 02683 . · arXiv
  23. Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., Sepassi, R., and Hechtman, B. Mesh-TensorFlow: Deep learning for supercomputers. In Neural Information Processing Systems , 2018.
  24. Trinh, T. H. and Le, Q. V. A simple method for commonsense reasoning. CoRR , abs/1806.02847, 2018. URL http://arxiv . org/abs/1806 . 02847 . · arXiv
  25. Turian, J., Ratinov, L., and Bengio, Y . Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , ACL '10, pp. 384-394, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.
  26. Valiant, L. G. A bridging model for parallel computation. Communications of the ACM , 33(8):103-111, 1990.
  27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR , abs/1706.03762, 2017. · arXiv
  28. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. ICLR , 2019.
  29. Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. URL http://arxiv . org/ abs/1906 . 08237 . · arXiv
  30. You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv:1708.03888 , 2017. · arXiv
  31. You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv:1904.00962 , 2019. · arXiv
  32. Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., and Choi, Y. Defending against neural fake news. CoRR , abs/1905.12616, 2019. URL http: //arxiv . org/abs/1905 . 12616 . · arXiv

Cited by

Sorted by citations

Authors
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B.
Affiliations
Nvidia (United Kingdom) (GB)
ORCID
Catanzaro, Bryan
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1909.08053
arXiv
1909.08053
OpenAlex
W2973727699
Cited by
893
References
14
Citations by year
2026: 20 · 2025: 94 · 2024: 186 · 2023: 168 · 2022: 105 · 2021: 181 · 2020: 127 · 2019: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, transformer models, multi-GPU training, memory constraints, PyTorch, BERT, GPT-2, training efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review