CoChat
Sign Up
PreprintCited by 2Stochastic Gradient Optimization TechniquesNeural Networks and ApplicationsDomain Adaptation and Few-Shot Learning

A Spectral Condition for Feature Learning

Open access
Yang, Greg, Simon, James B., Bernstein, Jeremy·2023·arXiv
DOI 10.48550/arxiv.2310.17813arXiv 2310.17813
Open this paper in CoChat — free accountOpen PDF

Abstract

The push to train ever larger neural networks has motivated the study of initialization and training at large network width. A key challenge is to scale training so that a network's internal representations evolve nontrivially at all widths, a process known as feature learning. Here, we show that feature learning is achieved by scaling the spectral norm of weight matrices and their updates like $\sqrt{\texttt{fan-out}/\texttt{fan-in}}$, in contrast to widely used but heuristic scalings based on Frobenius norm and entry size. Our spectral scaling analysis also leads to an elementary derivation of \emph{maximal update parametrization}. All in all, we aim to provide the reader with a solid conceptual understanding of feature learning in neural networks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.17813

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • P parameterization (Maximal Parameterization Update) allows hyperparameters such as learning rate tuned on small-sized models to be directly transferred to larger models.

    “Another commonly used optimization strategy is the P parameterization (Maximal Parameterization Update), which allows hyperparameters (e.g., learning rate) tuned on small-sized models to be directly transferred to larger models [142,143].”

    From A Survey of Large Language Models (2026) · cites this paper as [143]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Neural Information Processing Systems, 2019. Cited on page 1. Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: the silent alignment effect. In International Conference on Learning Representations, 2022. Cited on page 1.
  2. Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv:1607.06450, 2016. Cited on page 12. · arXiv
  3. Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signSGD: Compressed Optimisation for Non-Convex Problems, February 2018. URL https://arxiv.org/abs/1802.04434v3. Cited on page 7. · arXiv
  4. Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming-Yu Liu. On the distance between two neural networks and the stability of learning. In Neural Information Processing Systems, 2020a. Cited on pages 2, 6, 10, 12, and 13.
  5. Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Yu Liu, Anima Anandkumar, and Yisong Yue. Learning compositional functions via multiplicative weight updates. In Neural Information Processing Systems, 2020b. Cited on pages 10, 12, and 13.
  6. Jeremy Bernstein, Chris Mingard, Kevin Huang, Navid Azizan, and Yisong Yue. Automatic Gradient Descent: Deep Learning without Hyperparameters. arXiv:2304.05187, 2023. Cited on page 1. · arXiv
  7. Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. In Neural Information Processing Systems, 2022. Cited on page 10.
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Neural Information Processing Systems, 2020. Cited on page 1.
  9. Simon Carbonnelle and Christophe De Vleeschouwer. Layer rotation: A surprisingly simple indicator of generalization in deep networks? In ICML Workshop on Identifying and Understanding Deep Learning Phenomena, 2019. Cited on page 12.
  10. Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Neural Information Processing Systems, 2019. Cited on pages 10 and 12.
  11. Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-GPT: Open compute-optimal language models trained on the Cerebras wafer-scale cluster. arXiv:2304.03208, 2023. Cited on page 1. · arXiv
  12. Inderjit S. Dhillon and Joel A. Tropp. Matrix nearness problems with Bregman divergences. SIAM Journal on Matrix Analysis and Applications, 2008. Cited on page 2.
  13. Farzan Farnia, Jesse Zhang, and David Tse. Generalizable adversarial training via spectral normalization. In International Conference on Learning Representations, 2019. Cited on page 12.
  14. Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Neural Information Processing Systems, 2020. Cited on pages 1 and 12.
  15. Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020. Cited on page 12.
  16. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010. Cited on page 10.
  17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on ImageNet classification. In International Conference on Computer Vision, 2015. Cited on page 10.
  18. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015. Cited on page 12.
  19. Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Neural Information Processing Systems, 2018. Cited on pages 1, 10, 12, and 17.
  20. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Cited on page 1.
  21. Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks: Tricks of the Trade. Springer, 2002. Cited on pages 2 and 10.
  22. Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as Gaussian processes. In International Conference on Learning Represen- tations, 2018. Cited on page 1.
  23. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Neural Information Processing Systems, 2019. Cited on pages 1, 12, and 17.
  24. Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In Neural Information Processing Systems, 2020. Cited on page 1.
  25. Yang Liu, Jeremy Bernstein, Markus Meister, and Yisong Yue. Learning by turning: Neural architecture aware optimisation. In International Conference on Machine Learning, 2021. Cited on pages 2, 10, 12, and 13.
  26. Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: Dimension-free bounds and kernel limit. In Conference on Learning Theory, 2019. Cited on page 12.
  27. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for Generative Adversarial Networks. In International Conference on Learning Representations, 2018. Cited on page 12.
  28. Arkady S. Nemirovsky and David B. Yudin. Problem complexity and method efficiency in optimization. Wiley, 1983. Cited on page 2.
  29. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. Cited on page 2. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Neural Information Processing Systems, 2016. Cited on page 21.
  30. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv:2204.06125, 2022. Cited on page 1. · arXiv
  31. Grant M. Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 2022. Cited on page 12. Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: Extreme singular values. In International Congress of Mathematicians, 2010. Cited on pages 5 and 9.
  32. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, 2018. Cited on pages 2, 10, and 12.
  33. Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz, and Jaehoon Lee. On the infinite width limit of neural networks with a standard parameterization. arXiv:2001.07301, 2020. Cited on page 1. Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018. Cited on pages 5 and 9. · arXiv
  34. Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the NTK for understanding generalization in deep learning. arXiv:2206.10012, 2022. Cited on pages 1 and 12. · arXiv
  35. Sho Yaida. Meta-principled family of hyperparameter scaling strategies. arXiv:2210.04909, 2022. Cited on pages 6 and 12. · arXiv
  36. Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. In ICML 2021 Work- shop on Over-parameterization: Pitfalls and Opportunities, 2021a. https://slideslive.com/38963058/ feature-learning-in-infinitewidth-neural-networks. Cited on page 12.
  37. Greg Yang and Edward J. Hu. Tensor Programs IV: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, 2021b. Cited on pages 1, 3, 6, 12, and 18.
  38. Greg Yang and Etai Littwin. Tensor programs IVb: Adaptive optimization in the ∞-width limit. arXiv:2308.01814, 2023. Cited on page 18. · arXiv
  39. Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer. In Neural Information Processing Systems, 2021. Cited on pages 1, 10, and 13.
  40. Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32K for ImageNet training. Technical Report UCB/EECS-2017-156, University of California, Berkeley, 2017. Cited on pages 1, 2, 10, 12, and 13.
  41. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations, 2020. Cited on page 2.
  42. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, 2014. Cited on page 2.

Cited by

Sorted by citations

Authors
Yang, G., Simon, J. B., & Bernstein, J.
Affiliations
  • University of California, Berkeley (US)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.17813
arXiv
2310.17813
OpenAlex
W4388090547
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (1752814)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
representation learning, spectral norm, weight initialization, spectral scaling, gradient updates, internal representations, large-scale neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review