CoChat
Sign Up
PreprintCited by 2Parallel Computing and Optimization TechniquesAdvanced Neural Network ApplicationsTopic Modeling

The DOI matches its DataCite record.

Scalable Training of Mixture-of-Experts Models with Megatron Core

Open access
Yan, Zijie, Bai, Hongxiao, Yao, Xin, Liu, Dennis, Liu, Tong, Liu, Hongbin, Li, Pingtian, Wu, Evan, Fan, Shiqing, Tao, Li, Zhang, Robin, Wang, Yuzhong, Xu, Shifang, Chang, Jack, Chen, Xuwen, Li, Kunlun, Bai, Yan, Deng, Gao, Zheng, Nan, Korthikanti, Vijay Anand, Khattar, Abhinav, He, Ethan, Govande, Soham, Lym, Sangkug, Zhu, Zhongbo, Zhang, Qi, Yuan, Haochen, Ren, Xiaowei, Fu, Deyu, Ma, Tailai, Zhang, Shunkang, Shao, Jiang, Wang, Ray, Rengasamy, Vasudevan, Garg, Rachit, Bhavani, Santosh, Li, Xipeng, Zhou, Chandler, Wu, David, Wei, Yingcan, Aithal, Ashwath, Andersch, Michael, Shoeybi, Mohammad, Yao, Jiajie, Yang, June·2026·arXiv
DOI 10.48550/arxiv.2603.07685arXiv 2603.07685
Open this paper in CoChat — free accountOpen PDF

Abstract

Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.07685

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • KDA extends the delta-rule recurrence with a channel-wise forget gate.

    “KDA extends the delta-rule recurrence [105,138] with a channel-wise forget gate [63].”

    From Kimi K3: Open Frontier Intelligence (2026) · cites this paper as [138]

    The passages in §4.1.2 discuss MoE routing weights and memory-efficient permutation, not KDA or a delta-rule recurrence. Searches for “KDA” with “channel-wise forget gate” found no passage in our copy addressing the attributed claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 30

  1. [3]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, 'Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,' arXiv preprint arXiv:1701.06538 , 2017. 5, 9 · arXiv
  2. [7]W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, 'A survey on mixture of experts in large language models,' IEEE Transactions on Knowledge and Data Engineering , p. 1-20, 2025. 5, 6
  3. [9]R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, 'Adaptive mixtures of local experts,' Neural Computation , vol. 3, no. 1, pp. 79-87, 1991. 5
  4. [10]D. Eigen, M. Ranzato, and I. Sutskever, 'Learning factored representations in a deep mixture of experts,' arXiv preprint arXiv:1312.4314 , 2013. 5 · arXiv
  5. [11]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Advances in Neural Information Processing Systems , vol. 30, 2017. 5
  6. [12]D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, 'Gshard: Scaling giant models with conditional computation and automatic sharding,' arXiv preprint arXiv:2006.16668 , 2020. 5, 6, 10, 14, 16, 63, 64 · arXiv
  7. [14]C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Saab, J. Jose, R. Srivatsa, C. Wu, and Y. He, 'Tutel: Adaptive mixture-of-experts at scale,' in Proceedings of Machine Learning and Systems , vol. 5, 2023. 5, 10, 37
  8. [15]S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, 'Deepspeedmoe: Advancing mixture-of-experts inference and training to power next-generation ai scale,' in International Conference on Machine Learning , pp. 18332-18346, PMLR, 2022. 5, 37
  9. [16]D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. , 'Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models,' arXiv preprint arXiv:2401.06066 , 2024. 5 · arXiv
  10. [20]J. Krajewski, J. Ludziejewski, K. Adamczewski, M. Pióro, M. Krutul, S. Antoniak, K. Ciebiera, K. Król, T. Odrzygóźdź, P. Sankowski, et al. , 'Scaling laws for fine-grained mixture of experts,' arXiv preprint arXiv:2402.07871 , 2024. 5 · arXiv
  11. [24]T. Gale, D. Narayanan, C. Young, and M. Zaharia, 'MegaBlocks: Efficient Sparse Training with Mixtureof-Experts,' Proceedings of Machine Learning and Systems , vol. 5, 2023. 6, 10, 36
  12. [30]N. Shazeer, 'Glu variants improve transformer,' arXiv preprint arXiv:2002.05202 , 2020. 10 · arXiv
  13. [38]Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, 'Gpipe: Efficient training of giant neural networks using pipeline parallelism,' in Advances in Neural Information Processing Systems , vol. 32, 2019. 14, 32
  14. [39]D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Granger, P. B. Gibbons, and M. Zaharia, 'Pipedream: Generalized pipeline parallelism for dnn training,' in Proceedings of the 27th ACM Symposium on Operating Systems Principles , pp. 1-15, ACM, 2019. 14, 32
  15. [46]V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, 'Sequence parallelism: Long sequence training from system perspective,' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1842-1858, Association for Computational Linguistics, 2023. 14
  16. [49]S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y. He, and A. Bhatele, 'A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training,' in Proceedings of the 37th International Conference on Supercomputing , ICS '23, pp. 203-214, ACM, June 2023. 15
  17. [57]T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, '8-bit optimizers via block-wise quantization,' in International Conference on Learning Representations , 2022. 26
  18. [59]S. Wang, J. Wei, A. Sabne, A. Davis, R. Illikkal, S. Gangadharan, K. Wang, C. Liu, X. Cai, H. Xu, et al. , 'Overlap communication with dependent computation via decomposition in large deep learning models,' in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , pp. 93-106, 2022. 32
  19. [63]T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, 'TVM: An automated end-to-end optimizing compiler for deep learning,' in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , pp. 578-594, USENIX Association, 2018. 36
  20. [67]J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burber, G. Chauhan, A. Chourdia, et al. , 'PyTorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation,' in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , pp. 929-947, ACM, 2024. 39
  21. [68]P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, 'Mixed precision training,' in International Conference on Learning Representations , 2018. 48
  22. [77]A. Li, B. Liu, B. Hu, B. Li, B. Zeng, B. Ye, C. Tang, C. Tian, C. Huang, C. Zhang, et al. , 'Every activation boosted: Scaling general reasoner to 1 trillion open language foundation,' arXiv preprint arXiv:2510.22115 , 2025. 51 · arXiv
  23. [83]M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, 'BigBird: Transformers for longer sequences,' in Advances in Neural Information Processing Systems , vol. 33, pp. 17283-17297, 2020. 57
  24. [84]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, 'FlashAttention: Fast and memory-efficient exact attention with IO-awareness,' in Advances in Neural Information Processing Systems , 2022. 57
  25. [91]H. Ge, J. Feng, Q. Huang, F. Fu, X. Nie, L. Zuo, H. Lin, B. Cui, and X. Liu, 'Bytescale: Efficient scaling of llm training with a 2048k context length on more than 12,000 gpus,' arXiv preprint arXiv:2502.21231 , 2025. 61 · arXiv
  26. [92]Z. Wang, A. Cai, X. Xie, Z. Pan, Y. Guan, W. Chu, J. Wang, S. Li, J. Huang, C. Cai, et al. , 'Wlb-llm: Workload-balanced 4d parallelism for large language model training,' arXiv preprint arXiv:2503.17924 , 2025. 61 · arXiv
  27. [94]D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, 'Efficient large-scale language model training on gpu clusters using megatron-lm,' in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2021. 65
  28. [95]A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, 'Sparse upcycling: Training mixture-of-experts from dense checkpoints,' arXiv preprint arXiv:2212.05055 , 2022. 67 · arXiv
  29. [97]A. Vavre, E. He, D. Liu, Z. Yan, J. Yang, N. Tajbakhsh, and A. Aithal, 'Llama 3 meets moe: Efficient upcycling,' arXiv preprint arXiv:2412.09952 , 2024. 67 · arXiv
  30. [98]F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve, 'Better & faster large language models via multi-token prediction,' in Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024. 67

Cited by

Sorted by citations

Related

Metadata

Authors
Yan, Z., Bai, H., Yao, X., Liu, D., Liu, T., Liu, H., Li, P., Wu, E., Fan, S., Tao, L., Zhang, R., Wang, Y., Xu, S., Chang, J., Chen, X., Li, K., Bai, Y., Deng, G., Zheng, N., Korthikanti, V. A., Khattar, A., He, E., Govande, S., Lym, S., Zhu, Z., Zhang, Q., Yuan, H., Ren, X., Fu, D., Ma, T., Zhang, S., Shao, J., Wang, R., Rengasamy, V., Garg, R., Bhavani, S., Li, X., Zhou, C., Wu, D., Wei, Y., Aithal, A., Andersch, M., Shoeybi, M., Yao, J., & Yang, J.
Affiliations
Nvidia (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.07685
arXiv
2603.07685
OpenAlex
W7134822015
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
mixture of experts, large language model training, distributed training, memory optimization, model offloading, communication optimization, CUDA Graphs, NVIDIA GPUs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review