CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Muown: Row-Norm Control for Muon Optimization

Lion, Kai, Hübler, Florian, Li, Bingcong, Orvieto, Antonio, He, Niao·2026·arXiv
DOI 10.48550/arxiv.2605.10797arXiv 2605.10797
Open this paper in CoChat — free account

Abstract

Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the $\ell_\infty$ geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2605.10797

Cites

Statistics

Figures

Figure 1 Left two: Drift of the maximum row norm ∥ g t ∥ ∞ drives the growth of the spectral norm ∥ W t ∥ S ∞ for an MLP and attention output projection layer in a 500M transformer. When intervening by changing the parameterization s.t. the row magnitudes are fixed to the value at initialization, the systematic spectral norm increase vanishes. Right two: The remaining coherence factor λ max ( P t C t P t ) in the spectral norm exhibits no drift. λ denotes the weight decay, for more details, see Appendix A.4
Figure 1 Left two: Drift of the maximum row norm ∥ g t ∥ ∞ drives the growth of the spectral norm ∥ W t ∥ S ∞ for an MLP and attention output projection layer in a 500M transformer. When intervening by changing the parameterization s.t. the row magnitudes are fixed to the value at initialization, the systematic spectral norm increase vanishes. Right two: The remaining coherence factor λ max ( P t C t P t ) in the spectral norm exhibits no drift. λ denotes the weight decay, for more details, see Appendix A.4CC-BY · From the PDF
Figure 3 Perplexity as a function of learning rate for Muon, Muown, Lion, and SOAP on a 124M model. Muown achieves lower perplexity across all learning rates and is more robust to the choice of learning rate, maintaining strong performance even at higher values where Muon diverges. We repeat the average of two seeds, reporting min and max as shaded area. Details are reported in Appendix C.
Figure 3 Perplexity as a function of learning rate for Muon, Muown, Lion, and SOAP on a 124M model. Muown achieves lower perplexity across all learning rates and is more robust to the choice of learning rate, maintaining strong performance even at higher values where Muon diverges. We repeat the average of two seeds, reporting min and max as shaded area. Details are reported in Appendix C.CC-BY · From the PDF
Figure 2 Abstract Illustration of Muown and Muon, showing the parameterization (top), semantics and geometry (middle), and the optimizer step (bottom).
Figure 2 Abstract Illustration of Muown and Muon, showing the parameterization (top), semantics and geometry (middle), and the optimizer step (bottom).CC-BY · From the PDF
Figure 4 Left: Perplexity of Muon and Muown on a 2.7B transformer trained on FineWeb-Edu. Right: Ablation of weight decay and learning rate for Muon and Muown on 124M models after 5B tokens.
Figure 4 Left: Perplexity of Muon and Muown on a 2.7B transformer trained on FineWeb-Edu. Right: Ablation of weight decay and learning rate for Muon and Muown on 124M models after 5B tokens.CC-BY · From the PDF
Figure 5 Validation loss of Muon and Muown at three different model widths of 512, 768, 1280 with 12 layers each. We observe that Muown yields close to optimal validation loss for a wide range of learning rates from 2 -10 up to 2 -5 , irrespective of model width.
Figure 5 Validation loss of Muon and Muown at three different model widths of 512, 768, 1280 with 12 layers each. We observe that Muown yields close to optimal validation loss for a wide range of learning rates from 2 -10 up to 2 -5 , irrespective of model width.CC-BY · From the PDF
Figure 6 Normalized effective rank (Roy and Vetterli, 2007) of minibatch gradients for a 124M model. At a given training step, we sample 8 gradients and average their effective ranks.
Figure 6 Normalized effective rank (Roy and Vetterli, 2007) of minibatch gradients for a 124M model. At a given training step, we sample 8 gradients and average their effective ranks.CC-BY · From the PDF
Figure 7 Comparison of the noise coefficients for Muon and Muown. Shaded areas mark the interquartile range over weights.
Figure 7 Comparison of the noise coefficients for Muon and Muown. Shaded areas mark the interquartile range over weights.CC-BY · From the PDF
Figure 8 Learning rate and width ablation for Muon and Muown using the modded-nanoGPT codebase.
Figure 8 Learning rate and width ablation for Muon and Muown using the modded-nanoGPT codebase.CC-BY · From the PDF
Figure 9 Comparison of NorMuon and Muown on a 124M model.
Figure 9 Comparison of NorMuon and Muown on a 124M model.CC-BY · From the PDF
Figure 10 ∥ W t ∥ S ∞ for the layers in the 4th, 8th, and 12th transformer block of a 500M model.
Figure 10 ∥ W t ∥ S ∞ for the layers in the 4th, 8th, and 12th transformer block of a 500M model.CC-BY · From the PDF
Figure 11 ∥ g t ∥ ∞ for the layers in the 4th, 8th, and 12th transformer block of a 500M model.
Figure 11 ∥ g t ∥ ∞ for the layers in the 4th, 8th, and 12th transformer block of a 500M model.CC-BY · From the PDF
Figure 12 √ λ max ( P t C t P t ) for the layers in the 4th, 8th, and 12th transformer block of a 500M model.
Figure 12 √ λ max ( P t C t P t ) for the layers in the 4th, 8th, and 12th transformer block of a 500M model.CC-BY · From the PDF
Figure 13 Evolution of spectral norm ∥ W t ∥ S ∞ and maximum row norm ∥ g t ∥ ∞ over the course of training for linear layers of the 16th transformer block of a 500M model.
Figure 13 Evolution of spectral norm ∥ W t ∥ S ∞ and maximum row norm ∥ g t ∥ ∞ over the course of training for linear layers of the 16th transformer block of a 500M model.CC-BY · From the PDF
Figure 14 Histogram of singular values for linear layers of the 16th transformer block of a 500M model.
Figure 14 Histogram of singular values for linear layers of the 16th transformer block of a 500M model.CC-BY · From the PDF
Figure 15 Singular value distribution for linear layers of the 16th transformer block of a 500M model. attn refers to the attention block with w_q , w_k , w_v , w_out denoting query, key, value, and output projections, respectively. fc1 and fc2 refer to the first and second MLP layers of the block.
Figure 15 Singular value distribution for linear layers of the 16th transformer block of a 500M model. attn refers to the attention block with w_q , w_k , w_v , w_out denoting query, key, value, and output projections, respectively. fc1 and fc2 refer to the first and second MLP layers of the block.CC-BY · From the PDF
Figure 16 Normalized effective rank (Roy and Vetterli, 2007) of minibatch gradients for a 124M model. At a given training step, we sample 8 gradients and average their effective ranks.
Figure 16 Normalized effective rank (Roy and Vetterli, 2007) of minibatch gradients for a 124M model. At a given training step, we sample 8 gradients and average their effective ranks.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 45

  1. Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed Orthonormalized Updates, 2025. arXiv:2504.05295. · arXiv
  2. Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower Bounds for Non-Convex Stochastic Optimization. Mathematical Programming, 199(1): 165-214, 2023.
  3. Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The Geometry of Sign Gradient Descent, 2020. arXiv:2002.08056. · arXiv
  4. Amir Beck and Marc Teboulle. Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optimization. Operations Research Letters, 31(3):167-175, 2003.
  5. Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu. Advances in Optimizing Recurrent Networks. In IEEE Trans. Acoust., Speech, Sig. Process., 2013.
  6. Jeremy Bernstein and Laker Newhouse. Old Optimizer, New Norm: An Anthology. In OPT 2024: 16th Annual Workshop on Optimization for Machine Learning, 2024.
  7. Jeremy Bernstein and Laker Newhouse. Modular Duality in Deep Learning. In Proc. Int. Conf. on Machine Learning (ICML), 2025.
  8. Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed Optimisation for Non-Convex Problems. In Proc. Int. Conf. on Machine Learning (ICML), 2018.
  9. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An Open-Source Autoregressive Language Model. In Proceedings of BigScience Episode #5 -Workshop on Challenges & Perspectives in Creating Large Language Models. ACL, 2022.
  10. Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization Methods for Large-scale Machine Learning. SIAM review, 60(2):223-311, 2018.
  11. Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
  12. David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic Spectral Descent for Restricted Boltzmann Machines. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), 2015.
  13. Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion Secretly Solves Constrained Optimiza- tion: As Lyapunov Predicts. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
  14. Lizhang Chen, Jonathan Li, and Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. In OPT 2025: 17th Annual Workshop on Optimization for Machine Learning, 2025.
  15. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. Symbolic Discovery of Optimization Algorithms. In Proc. Neural Information Processing Systems (NeurIPS), 2023.
  16. Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval Networks: Improving Robustness to Adversarial Examples. In Proc. Int. Conf. on Machine Learning (ICML), 2017.
  17. Ashok Cutkosky and Harsh Mehta. Momentum Improves Normalized SGD. In Proc. Int. Conf. on Machine Learning (ICML), 2020.
  18. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned Stochastic Tensor Optimization. In Proc. Int. Conf. on Machine Learning (ICML), 2018.
  19. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training Compute-optimal Large Language Models. In Proc. Neural Information Processing Systems (NeurIPS), 2022.
  20. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In Proc. Conf. on Language Modeling (COLM), 2024.
  21. Kimi-Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, and Yanru Chen et al. Kimi K2: Open Agentic Intelligence, 2025. arXiv:2507.20534. · arXiv
  22. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. Int. Conf. on Learning Representations (ICLR), 2015.
  23. Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be. In Proc. Int. Conf. on Learning Representations (ICLR), 2023.
  24. Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable Optimization in the Modular Norm. In Proc. Neural Information Processing Systems (NeurIPS), 2024.
  25. Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making Muon More Efficient and Scalable, 2025. arXiv:2510.05491. · arXiv
  26. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is Scalable for LLM Training, 2025. arXiv:2502.16982. · arXiv
  27. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
  28. James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-factored Approx- imate Curvature. In Proc. Int. Conf. on Machine Learning (ICML), 2015.
  29. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks, 2018. arXiv:1802.05957. · arXiv
  30. Arkadi Nemirovski and David B. Yudin. Problem Complexity and Method Efficiency in Optimization. Wiley-Interscience Series in Discrete Mathematics. Wiley-Interscience, 1983.
  31. Yu.E. Nesterov. A Method for Solving the Convex Programming Problem with Convergence Rate $O(1/K^2)$. Doklady Akademii Nauk SSSR, 269(3):543-547, 1983. ISSN 0002-3264.
  32. Laker Newhouse, R. Preston Hess, Franz Cesista, Andrii Zahorodnii, Jeremy Bernstein, and Phillip Isola. Training Transformers with Enforced Lipschitz Constants, 2025. arXiv:2507.13338. · arXiv
  33. Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training Deep Learning Models with Norm-Constrained LMOs. In Proc. Int. Conf. on Machine Learning (ICML), 2025.
  34. Olivier Roy and Martin Vetterli. The Effective Rank: A Measure of Effective Dimensionality. In 15th European Signal Processing Conference, pages 606-610. IEEE, 2007.
  35. Tim Salimans and Durk P Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Proc. Neural Information Processing Systems (NeurIPS), 2016.
  36. Maria-Eleni Sfyraki and Jun-Kun Wang. Lions and Muons: Optimization via Stochastic Frank- Wolfe, 2025. arXiv:2506.04192. · arXiv
  37. Noam Shazeer. GLU Variants Improve Transformer, 2020. arXiv:2002.05202. · arXiv
  38. Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the Convergence Analysis of Muon, 2025. arXiv:2505.23737. · arXiv
  39. Mark Tuddenham, Adam Prügel-Bennett, and Jonathan Hare. Orthogonalising gradients to speed up neural network optimisation, 2022. arXiv:2202.07052. · arXiv
  40. Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and Stabilizing Shampoo using Adam. In Proc. Int. Conf. on Learning Representations (ICLR), 2025.
  41. Shuo Xie and Zhiyuan Li. Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization, 2024. arXiv:2404.04454. · arXiv
  42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report, 2024a. arXiv:2407.10671. · arXiv
  43. Greg Yang, James B. Simon, and Jeremy Bernstein. A Spectral Condition for Feature Learning, 2024b. arXiv:2310.17813. · arXiv
  44. Yuichi Yoshida and Takeru Miyato. Spectral Norm Regularization for Improving the Generaliz- ability of Deep Learning, 2017. arXiv:1705.10941. · arXiv
  45. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Proc. Neural Information Processing Systems (NeurIPS), 2019.

Cited by

  • MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

    Core Team, Qiao, Z., Hua, Z., Ou, Z., Yue, Z., Jiang, Z., Huang, Z., Chen, Z., Zheng, Z., Xu, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.11959 · Cited by 9

    “…of Muon optimizer, Muown, for the hidden weight matrices during mid-training to prepare the model for subsequent large-batch RL. Muown augments Muon with explicit row-norm control, mitigating spectral-norm drift and reducing sensitivity to weight decay at negligible overhead (Lion et al., 2026).”

See all 1

Metadata

Authors
Lion, K., Hübler, F., Li, B., Orvieto, A., & He, N.
Venue
arXiv
Publisher
arXiv
Published
2026
DOI
10.48550/arxiv.2605.10797
arXiv
2605.10797
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Muown: Row-Norm Control for Muon Optimization. https://app.cochat.ai/papers/10.48550/arxiv.2605.10797
The paper
Lion, K., Hübler, F., Li, B., Orvieto, A., & He, N. (2026). Muown: Row-Norm Control for Muon Optimization. arXiv. https://doi.org/10.48550/arxiv.2605.10797