CoChat
Sign Up
PreprintCited by 28Machine Learning and Data ClassificationDomain Adaptation and Few-Shot LearningStochastic Gradient Optimization Techniques

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Open access
Yang, Greg, Hu, Edward J., Babuschkin, Igor, Sidor, Szymon, Liu, Xiaodong, Farhi, David, Ryder, Nick, Pachocki, Jakub +2 more·2022·arXiv
DOI 10.48550/arxiv.2203.03466arXiv 2203.03466
Open this paper in CoChat — free accountOpen PDF

Abstract

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.03466

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • P parameterization (Maximal Parameterization Update) allows hyperparameters such as learning rate tuned on small-sized models to be directly transferred to larger models.

    “Another commonly used optimization strategy is the P parameterization (Maximal Parameterization Update), which allows hyperparameters (e.g., learning rate) tuned on small-sized models to be directly transferred to larger models [142,143].”

    From A Survey of Large Language Models (2026) · cites this paper as [142]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 50

  1. [5][5] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. ReZero is All You Need: Fast Convergence at Large Depth. arXiv:2003.04887 [cs, stat] , June 2020. URL http://arxiv.org/abs/2003.04887 . - · arXiv
  2. [6][6] Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming-Yu Liu. On the distance between two neural networks and the stability of learning. arXiv:2002.03432 [cs, math, stat] , January 2021. URL http://arxiv.org/abs/2002.03432 . - · arXiv
  3. [8][8] Simon Carbonnelle and Christophe De Vleeschouwer. Layer rotation: a surprisingly powerful indicator of generalization in deep networks? arXiv:1806.01603 [cs, stat] , July 2019. URL http://arxiv.org/abs/1806.01603 . - · arXiv
  4. [11][11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs] , May 2019. URL http://arxiv.org/abs/1810.04805 . - · arXiv
  5. [12][12] Xiaohan Ding, Chunlong Xia, Xiangyu Zhang, Xiaojie Chu, Jungong Han, and Guiguang Ding. RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition. arXiv:2105.01883 [cs] , August 2021. URL http://arxiv.org/abs/2105.01883 . - · arXiv
  6. [13][13] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 249-256, Chia Laguna Resort, Sardinia, Italy, May 2010. PMLR. URL http://proceedings.mlr.press/v9/glorot10a.html . -
  7. [14][14] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv:1705.08741 [cs, stat] , May 2017. URL http://arxiv.org/abs/1705.08741 . - · arXiv
  8. [15][15] Samuel Horváth, Aaron Klein, Peter Richtárik, and Cédric Archambeau. Hyperparameter transfer learning with adaptive complexity. CoRR , abs/2102.12810, 2021. URL https: //arxiv.org/abs/2102.12810 . - · arXiv
  9. [17][17] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. arXiv:1806.07572 [cs, math, stat] , June 2018. URL http://arxiv.org/abs/1806.07572 . - · arXiv
  10. [20][20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. - · arXiv
  11. [21][21] Jaehoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl-dickstein. Deep Neural Networks as Gaussian Processes. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=B1EA-M-0Z . -
  12. [23][23] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. JMLR 18 , page 52. -
  13. [24][24] Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay Attention to MLPs. arXiv:2105.08050 [cs] , June 2021. URL http://arxiv.org/abs/2105.08050 . - · arXiv
  14. [26][26] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the Difficulty of Training Transformers. arXiv:2004.08249 [cs, stat] , September 2020. URL http://arxiv.org/abs/2004.08249 . - · arXiv
  15. [27][27] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4487-4496, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ P19-1441 . -
  16. [28][28] Yang Liu, Jeremy Bernstein, Markus Meister, and Yisong Yue. Learning by Turning: Neural Architecture Aware Optimisation. arXiv:2102.07227 [cs] , September 2021. URL http: //arxiv.org/abs/2102.07227 . - · arXiv
  17. [29][29] Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaussian Process Behaviour in Wide Deep Neural Networks. arXiv:1804.11271 [cs, stat] , April 2018. URL http://arxiv.org/abs/1804.11271 . arXiv: 1804.11271. - · arXiv
  18. [30][30] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An Empirical Model of Large-Batch Training. arXiv:1812.06162 [cs, stat] , December 2018. URL http: //arxiv.org/abs/1812.06162 . - · arXiv
  19. [31][31] Luke Melas-Kyriazi. Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet. arXiv:2105.02723 [cs] , May 2021. URL http://arxiv. org/abs/2105.02723 . - · arXiv
  20. [33][33] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling, mit license. In Proceedings of NAACL-HLT 2019: Demonstrations , 2019. -
  21. [34][34] Daniel S. Park, Jascha Sohl-Dickstein, Quoc V. Le, and Samuel L. Smith. The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study. May 2019. URL https://arxiv.org/abs/1905.03776v1 . - · arXiv
  22. [35][35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, bsd-style license. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf . -
  23. [36][36] Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable Hyperparameter Transfer Learning. NeurIPS 2018 , page 11. -
  24. [37][37] Martin Popel and Ondˇ rej Bojar. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics , 110(1):43-70, April 2018. ISSN 1804-0462. doi: 10.2478/ pralin-2018-0002. URL http://content.sciendo.com/view/journals/pralin/110/ 1/article-p43.xml . -
  25. [38][38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs, stat] , July 2020. URL http://arxiv.org/ abs/1910.10683 . - · arXiv
  26. [39][39] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A Constructive Prediction of the Generalization Error Across Scales. arXiv:1909.12673 [cs, stat] , December 2019. URL http://arxiv.org/abs/1909.12673 . - · arXiv
  27. [40][40] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. arXiv:1611.01232 [cs, stat] , November 2016. URL http://arxiv.org/ abs/1611.01232 . - · arXiv
  28. [41][41] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism on Neural Network Training. arXiv:1811.03600 [cs, stat] , November 2018. URL http://arxiv.org/abs/1811.03600 . - · arXiv
  29. [42][42] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. April 2018. URL https://arxiv.org/abs/1804.04235v1 . - · arXiv
  30. [43][43] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR , abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053 . - · arXiv
  31. [44][44] Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't Decay the Learning Rate, Increase the Batch Size. arXiv:1711.00489 [cs, stat] , November 2017. URL http://arxiv. org/abs/1711.00489 . - · arXiv
  32. [47][47] Danny Stoll, Jörg K.H. Franke, Diane Wagner, Simon Selg, and Frank Hutter. Hyperparameter transfer across developer adjustments, 2021. URL https://openreview.net/forum?id= WPO0vDYLXem . -
  33. [48][48] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An all-MLP Architecture for Vision. arXiv:2105.01601 [cs] , June 2021. URL http://arxiv.org/abs/2105.01601 . - · arXiv
  34. [49][49] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward networks for image classification with data-efficient training. arXiv:2105.03404 [cs] , June 2021. URL http://arxiv.org/abs/2105.03404 . - · arXiv
  35. [50][50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR , abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762 . - · arXiv
  36. [51][51] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018 , page 353, 2018. -
  37. [52][52] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/N18-1101 . -
  38. [53][53] Greg Yang. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. arXiv:1910.12478 [cond-mat, physics:math-ph] , December 2019. URL http://arxiv.org/abs/1910.12478 . - · arXiv
  39. [54][54] Greg Yang. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation. arXiv:1902.04760 [cond-mat, physics:math-ph, stat] , February 2019. URL http://arxiv.org/abs/1902. 04760 . - · arXiv
  40. [55][55] Greg Yang. Tensor Programs II: Neural Tangent Kernel for Any Architecture. arXiv:2006.14548 [cond-mat, stat] , August 2020. URL http://arxiv.org/abs/2006.14548 . - · arXiv
  41. [56][56] Greg Yang. Tensor Programs III: Neural Matrix Laws. arXiv:2009.10685 [cs, math] , September 2020. URL http://arxiv.org/abs/2009.10685 . - · arXiv
  42. [57][57] Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. arXiv , 2020. -
  43. [58][58] Greg Yang and Etai Littwin. Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics. arXiv:2105.03703 [cs, math] , May 2021. URL http: //arxiv.org/abs/2105.03703 . - · arXiv
  44. [59][59] Greg Yang and Sam S. Schoenholz. Deep Mean Field Theory: Layerwise Variance and Width Variation as Methods to Control Gradient Explosion. February 2018. URL https: //openreview.net/forum?id=rJGY8GbR-. -
  45. [60][60] Greg Yang and Samuel S. Schoenholz. Mean Field Residual Networks: On the Edge of Chaos. arXiv:1712.08969 [cond-mat, physics:nlin] , December 2017. URL http://arxiv.org/abs/ 1712.08969 . - · arXiv
  46. [61][61] Greg Yang, Michael Santacroce, and Edward J Hu. Efficient computation of deep nonlinear infinite-width neural networks that learn features. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=tUMr0Iox8XW . -
  47. [62][62] Dani Yogatama and Gideon Mann. Efficient Transfer Learning Method for Automatic Hyperparameter Tuning. In Artificial Intelligence and Statistics , pages 1077-1085. PMLR, April 2014. URL http://proceedings.mlr.press/v33/yogatama14.html . -
  48. [63][63] Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv:1708.03888 [cs] , September 2017. URL http://arxiv.org/abs/1708.03888 . - · arXiv
  49. [64][64] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. arXiv:1904.00962 [cs, stat] , January 2020. URL http://arxiv.org/abs/1904.00962 . - · arXiv
  50. [66][66] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual Learning Without Normalization via Better Initialization. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1gsz30cKX .

Cited by

Sorted by citations

Related

Authors
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., & Gao, J.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2203.03466
arXiv
2203.03466
OpenAlex
W4221164116
Cited by
23
References
0
Citations by year
2025: 6 · 2024: 11 · 2023: 4 · 2022: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
hyperparameter optimization, large-scale neural networks, model size scaling, transformer models, ResNet, BERT, GPT-3, training cost reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review