CoChat
Sign Up
PreprintCited by 9Stochastic Gradient Optimization TechniquesTopic ModelingNatural Language Processing Techniques

Muon is scalable for llm training

Open access
Liu, Jingyuan, Su, Jianlin, Yao, Xingcheng, Jiang, Zhejun, Lai, Guokun, Du, Yulun, Qin, Yidao, Xu, Weixin +20 more·2025·arXiv
DOI 10.48550/arxiv.2502.16982arXiv 2502.16982
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.16982

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Shampoo and Muon share similar designs in their update rules, and Muon introduces Newton-Schulz iterations to avoid the expensive SVD computation.

    “Shampoo and Muon share similar designs in their update rules, and Muon introduces Newton-Schulz iterations to avoid the expensive SVD computation.”

    From A Survey of Large Language Models (2026) · cites this paper as [148]

    Section 2.1 (p. 2) says Muon uses Newton–Schulz iterations to approximate an orthogonalized update expressed through SVD, supporting the point that the update need not compute an SVD. Our copy does not describe Shampoo or compare its update rule with Muon’s, so the claimed similarity is unsupported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • MuSGD is a hybrid Muon-SGD optimizer adapted from large language model training.

    “MuSGD, a hybrid MuonSGD optimizer adapted from large language model training”

    From Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models (2026) · cites this paper as [40]

    The Introduction (p. 2) and Conclusions (p. 11) establish that this paper develops Muon for large-scale language-model training. But the Discussions (p. 11) describe its hybrid implementation as Muon used with Adam, not SGD; our copy does not identify an optimizer called MuSGD. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Unlike Adam, optimizers based on gradient spectral norms such as Muon require full matrix representations, which necessitates additional parameter sharding strategies in their corresponding ZeRO implementations.

    “Note that, unlike Adam, optimizers based on gradient spectral norms (e.g., Muon) require full matrix representations, which necessitates additional parameter sharding strategies in their corresponding ZeRO implementations [148].”

    From A Survey of Large Language Models (2026) · cites this paper as [148]

  • A new class of spectral-norm based optimizers has been increasingly adopted in large-scale training, such as Shampoo, SOAP, and Muon.

    “Recently, a new class of spectral-norm based optimizers has been increasingly adopted in large-scale training (e.g., Shampoo [146], SOAP [147], and Muon [117,148]).”

    From A Survey of Large Language Models (2026) · cites this paper as [148]

  • Muon optimizer has demonstrated roughly 2× computational efficiency over AdamW in large language model training.

    “Recent work on the Muon optimizer [40] has demonstrated roughly 2 × computational efficiency over AdamW in large language model training”

    From Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models (2026) · cites this paper as [40]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 3× · see citing works

    “Hence, we perform a grid search for hyper-parameters of AdamW, following the compute-optimal training setup (Kaplan et al. 2020) (the grid search experiments can be found in Appendix B).”

Flagged first, then by citations

Research works printed but not linked: 38

  1. Alter, Orly, Patrick O. Brown, and David Botstein. 'Singular value decomposition for genome-wide expression data processing and modeling'. In: Proceedings of the National Academy of Sciences 97.18 (2000), pp. 10101-10106. DOI: 10.1073/pnas.97.18.10101 . eprint: https://www.pnas.org/doi/pdf/10.1073/pnas.97.18.10101 . URL: https://www.pnas.org/doi/abs/10.1073/pnas.97.18.10101 . · DOI
  2. Austin, Jacob et al. Program Synthesis with Large Language Models . 2021. arXiv: 2108.07732 [cs.PL] . URL: https://arxiv.org/abs/2108.07732 . · arXiv
  3. Bernstein, Jeremy and Laker Newhouse. Old Optimizer, New Norm: An Anthology . 2024. arXiv: 2409.20325 [cs.LG] . URL: https://arxiv.org/abs/2409.20325 . · arXiv
  4. Bi, Xiao et al. 'Deepseek llm: Scaling open-source language models with longtermism'. In: arXiv preprint arXiv:2401.02954 (2024). · arXiv
  5. Chen, Mark et al. 'Evaluating Large Language Models Trained on Code'. In: (2021). arXiv: 2107.03374 [cs.LG] . · arXiv
  6. Cobbe, Karl et al. Training Verifiers to Solve Math Word Problems . 2021. arXiv: 2110.14168 [cs.LG] . URL: https://arxiv.org/abs/2110.14168 . · arXiv
  7. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model . 2024. arXiv: 2405.04434 [cs.CL] . · arXiv
  8. DeepSeek-AI et al. DeepSeek-V3 Technical Report . 2024. arXiv: 2412.19437 [cs.CL] . URL: https://arxiv.org/ abs/2412.19437 . · arXiv
  9. Grattafiori, Aaron et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI] . URL: https://arxiv. org/abs/2407.21783 . · arXiv
  10. Hendrycks, Dan, Collin Burns, Steven Basart, et al. Measuring Massive Multitask Language Understanding . 2021. arXiv: 2009.03300 [cs.CY] . URL: https://arxiv.org/abs/2009.03300 . · arXiv
  11. Hendrycks, Dan, Collin Burns, Saurav Kadavath, et al. Measuring Mathematical Problem Solving With the MATH Dataset . 2021. arXiv: 2103.03874 [cs.LG] . URL: https://arxiv.org/abs/2103.03874 . · arXiv
  12. Hoffmann, Jordan et al. Training Compute-Optimal Large Language Models . 2022. arXiv: 2203.15556 [cs.CL] . URL: https://arxiv.org/abs/2203.15556 . · arXiv
  13. Huang, Yuzhen et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models . 2023. arXiv: 2305.08322 [cs.CL] . URL: https://arxiv.org/abs/2305.08322 . · arXiv
  14. Joshi, Mandar et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension . 2017. arXiv: 1705.03551 [cs.CL] . URL: https://arxiv.org/abs/1705.03551 . · arXiv
  15. Kingma, Diederik P. and Jimmy Ba. 'Adam: A Method for Stochastic Optimization'. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings . Ed. by Yoshua Bengio and Yann LeCun. 2015. URL: http://arxiv.org/abs/1412.6980 . · arXiv
  16. Li, Haonan et al. CMMLU:Measuring massive multitask language understanding in Chinese . 2024. arXiv: 2306.09212 [cs.CL] . URL: https://arxiv.org/abs/2306.09212 . · arXiv
  17. Li, Xi-Lin. 'Preconditioned Stochastic Gradient Descent'. In: IEEE Transactions on Neural Networks and Learning Systems 29.5 (May 2018), pp. 1454-1466. ISSN: 2162-2388. DOI: 10.1109/tnnls.2017.2672978 . URL: http: //dx.doi.org/10.1109/TNNLS.2017.2672978 . · DOI
  18. -Preconditioner on Matrix Lie Group for SGD . 2018. arXiv: 1809.10232 [stat.ML] . URL: https://arxiv.org/ abs/1809.10232 . · arXiv
  19. -Stochastic Hessian Fittings with Lie Groups . 2024. arXiv: 2402.11858 [stat.ML] . URL: https://arxiv.org/ abs/2402.11858 . · arXiv
  20. Li, Xilin. Black Box Lie Group Preconditioners for SGD . 2022. arXiv: 2211.04422 [stat.ML] . URL: https: //arxiv.org/abs/2211.04422 . · arXiv
  21. Liu, Hong et al. 'Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training'. In: The Twelfth International Conference on Learning Representations . 2024. URL: https://openreview.net/forum? id=3xHDeA8Noi .
  22. Loshchilov, Ilya and Frank Hutter. 'Decoupled Weight Decay Regularization'. In: International Conference on Learning Representations . 2019. URL: https://openreview.net/forum?id=Bkg6RiCqY7 .
  23. OLMo, Team et al. '2 OLMo 2 Furious'. In: arXiv preprint arXiv:2501.00656 (2024). · arXiv
  24. OpenAI et al. GPT-4 Technical Report . 2024. arXiv: 2303.08774 [cs.CL] . URL: https://arxiv.org/abs/2303. 08774 . · arXiv
  25. Pethick, Thomas et al. Training Deep Learning Models with Norm-Constrained LMOs . 2025. arXiv: 2502.07529 [cs.LG] . URL: https://arxiv.org/abs/2502.07529 . · arXiv
  26. Pooladzandi, Omead and Xi-Lin Li. Curvature-Informed SGD via General Purpose Lie-Group Preconditioners . 2024. arXiv: 2402.04553 [cs.LG] . URL: https://arxiv.org/abs/2402.04553 . · arXiv
  27. Rajbhandari, Samyam et al. 'ZeRO: Memory optimizations Toward Training Trillion Parameter Models'. In: (Nov. 2020), pp. 1-16. DOI: 10.1109/sc41405.2020.00024 . URL: http://dx.doi.org/10.1109/SC41405.2020. 00024 . · DOI
  28. Roy, Olivier and Martin Vetterli. 'The effective rank: A measure of effective dimensionality'. In: 2007 15th European Signal Processing Conference . 2007, pp. 606-610.
  29. Shazeer, Noam. Fast Transformer Decoding: One Write-Head is All You Need . 2019. arXiv: 1911.02150 [cs.NE] . URL: https://arxiv.org/abs/1911.02150 . · arXiv
  30. Shoeybi, Mohammad et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism . 2020. arXiv: 1909.08053 [cs.CL] . URL: https://arxiv.org/abs/1909.08053 . · arXiv
  31. Suzgun, Mirac et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them . 2022. arXiv: 2210.09261 [cs.CL] . URL: https://arxiv.org/abs/2210.09261 . · arXiv
  32. Team, Gemini et al. Gemini: A Family of Highly Capable Multimodal Models . 2024. arXiv: 2312.11805 [cs.CL] . URL: https://arxiv.org/abs/2312.11805 . · arXiv
  33. Team, Gemma et al. 'Gemma 2: Improving open language models at a practical size'. In: arXiv preprint arXiv:2408.00118 (2024). · arXiv
  34. Vyas, Nikhil et al. 'SOAP: Improving and Stabilizing Shampoo using Adam'. In: The Thirteenth International Conference on Learning Representations . 2025. URL: https://openreview.net/forum?id=IDxZhXrpNf .
  35. Wang, Yubo et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark . 2024. arXiv: 2406.01574 [cs.CL] . URL: https://arxiv.org/abs/2406.01574 . · arXiv
  36. Wei, Tianwen et al. CMATH: Can Your Language Model Pass Chinese Elementary School Math Test? 2023. arXiv: 2306.16636 [cs.CL] . URL: https://arxiv.org/abs/2306.16636 . · arXiv
  37. Yang, An et al. 'Qwen2.5 Technical Report'. In: arXiv preprint arXiv:2412.15115 (2024). · arXiv
  38. Yuan, Huizhuo et al. MARS: Unleashing the Power of Variance Reduction for Training Large Models . 2024. arXiv: 2411.10438 [cs.LG] . · arXiv

Cited by

Sorted by citations

Authors
Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., Chen, Y., Zheng, H., Liu, Y., Liu, S., Yin, B., He, W., Zhu, H., Wang, Y., Wang, J., Dong, M., Zhang, Z., Kang, Y., Zhang, H., Xu, X., Zhang, Y., Wu, Y., Zhou, X., & Yang, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.16982
arXiv
2502.16982
OpenAlex
W4414846785
Cited by
2
References
1
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
Muon optimizer, large language model training, weight decay, moonlight, mixture of experts, training efficiency, distributed training, memory efficiency, communication efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review