CoChat
Sign Up
PreprintCited by 51Stochastic Gradient Optimization TechniquesNeural Networks and ApplicationsGaussian Processes and Bayesian Inference

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

On the linearity of large non-linear models: when and why the tangent kernel is constant

Open access
Liu, Chaoyue, Zhu, Libin, Belkin, Mikhail·2020·arXiv
DOI 10.48550/arxiv.2010.01092arXiv 2010.01092
Open this paper in CoChat — free accountOpen PDF

Abstract

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.01092

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 21

  1. [1][1] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. 'A Convergence Theory for Deep Learning via Over-Parameterization'. In: International Conference on Machine Learning . 2019, pp. 242252. -
  2. [2][2] Shun-ichi Amari. 'Any target function exists in a neighborhood of any sufficiently wide random network: A geometrical perspective'. In: arXiv preprint arXiv:2001.06931 (2020). - · arXiv
  3. [3][3] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. 'Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks'. In: International Conference on Machine Learning . 2019, pp. 322-332. -
  4. [4][4] Peter L Bartlett, David P Helmbold, and Philip M Long. 'Gradient descent with identity initialization efficiently learns positive-definite linear transformations by deep residual networks'. In: Neural computation 31.3 (2019), pp. 477-502. -
  5. [5][5] Anthony Carbery and James Wright. 'Distributional and L q norm inequalities for polynomials over convex bodies in R n '. In: Mathematical research letters 8.3 (2001), pp. 233-248. -
  6. [6][6] Lenaic Chizat, Edouard Oyallon, and Francis Bach. 'On lazy training in differentiable programming'. In: Advances in Neural Information Processing Systems . 2019, pp. 2933-2943. -
  7. [7][7] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. 'Gradient Descent Finds Global Minima of Deep Neural Networks'. In: International Conference on Machine Learning . 2019, pp. 1675-1685. -
  8. [8][8] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. 'Gradient descent provably optimizes over-parameterized neural networks'. In: arXiv preprint arXiv:1810.02054 (2018). - · arXiv
  9. [10][10] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. 'Linearized two-layers neural networks in high dimension'. In: arXiv preprint arXiv:1904.12191 (2019). - · arXiv
  10. [12][12] Arthur Jacot, Franck Gabriel, and Clément Hongler. 'Neural tangent kernel: Convergence and generalization in neural networks'. In: Advances in neural information processing systems . 2018, pp. 8571-8580. -
  11. [13][13] Ziwei Ji and Matus Telgarsky. 'Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks'. In: arXiv preprint arXiv:1909.12292 (2019). - · arXiv
  12. [14][14] Beatrice Laurent and Pascal Massart. 'Adaptive estimation of a quadratic functional by model selection'. In: Annals of Statistics (2000), pp. 1302-1338. -
  13. [15][15] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. 'Efficient backprop'. In: Neural networks: Tricks of the trade . Springer, 2012, pp. 9-48. -
  14. [16][16] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. 'Wide neural networks of any depth evolve as linear models under gradient descent'. In: Advances in neural information processing systems . 2019, pp. 8570-8581. -
  15. [17][17] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. 'Toward a theory of optimization for overparameterized systems of non-linear equations: the lessons of deep learning'. In: arXiv preprint arXiv:2003.00307 (2020). - · arXiv
  16. [18][18] Prajit Ramachandran, Barret Zoph, and Quoc V Le. 'Searching for activation functions'. In: arXiv preprint arXiv:1710.05941 (2017). - · arXiv
  17. [19][19] Takashi Sakai. 'On Riemannian manifolds admitting a function whose gradient is of constant norm'. In: Kodai Mathematical Journal 19.1 (1996), pp. 39-51. -
  18. [20][20] Ruoyu Sun. 'Optimization for deep learning: theory and algorithms'. In: arXiv preprint arXiv:1912.08957 (2019). - · arXiv
  19. [21][21] Terence Tao. Topics in random matrix theory . Vol. 132. American Mathematical Soc., 2012. -
  20. [22][22] Roman Vershynin. 'Introduction to the non-asymptotic analysis of random matrices'. In: arXiv preprint arXiv:1011.3027 (2010). - · arXiv
  21. [23][23] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. 'Stochastic gradient descent optimizes over-parameterized deep relu networks'. In: arXiv preprint arXiv:1811.08888 (2018). · arXiv

Cited by

Sorted by citations

  • How Do Vision Transformers Work?

    Park, N., & Kim, S. (2022) · arXiv · doi:10.48550/arxiv.2202.06709 · Cited by 356

    “Neural tangent kernel (NTK) (Jacot et al., 2018) theoretically implies that the loss landscape of a ViT is convex and flat when the number of heads or the number of embedding dimensions per head goes to infinity (Hron et al., 2020; Liu et al., 2020).”
    BackgroundMethod

See all 51

Metadata

Authors
Liu, C., Zhu, L., & Belkin, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
15954-15964
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.01092
arXiv
2010.01092
OpenAlex
W3089353926
Cited by
51
References
1
Citations by year
2025: 6 · 2024: 12 · 2023: 9 · 2022: 7 · 2021: 17
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
Keywords
neural tangent kernel, wide neural networks, lazy training, gradient descent, nonlinear models, network width

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review