CoChat
Sign Up
PreprintCited by 1,562Gaussian Processes and Bayesian InferenceNeural Networks and ApplicationsStochastic Gradient Optimization Techniques

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Open access
Jacot, Arthur, Gabriel, Franck, Hongler, Clément·2018·arXiv
DOI 10.48550/arxiv.1806.07572arXiv 1806.07572
Open this paper in CoChat — free accountOpen PDF

Abstract

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_θ$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_θ$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.07572

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • NTK theoretically implies that the loss landscape of a ViT is convex and flat when the number of heads or embedding dimensions per head goes to infinity.

    “Neural tangent kernel (NTK) (Jacot et al., 2018) theoretically implies that the loss landscape of a ViT is convex and flat when the number of heads or the number of embedding dimensions per head goes to infinity (Hron et al., 2020; Liu et al., 2020).”

    From How Do Vision Transformers Work? (2022) · cites this paper as Jacot et al., 2018

  • Bischke et al.'s cascaded multi-task loss for preserving semantic segmentation boundaries in satellite imagery outperformed state-of-the-art techniques by 8.3% without extra post-processing.

    “Their method outperformed the state-of-the-art techniques by 8.3% without an extra post-processing step.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [531]

    The Introduction (p. 1) and Conclusion (p. 9) discuss neural-network training and the Neural Tangent Kernel, not satellite-image segmentation. Searches for Bischke, the 8.3% improvement, and post-processing found no passage in our copy addressing the claimed result. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 21

  1. [1][1] M. Belkin, S. Ma, and S. Mandal. To understand deep learning we need to understand kernel learning. arXiv preprint , Feb 2018. -
  2. [2][2] Y. Cho and L. K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems 22 , pages 342-350. Curran Associates, Inc., 2009. -
  3. [3][3] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The Loss Surfaces of Multilayer Networks. Journal of Machine Learning Research , 38:192-204, nov 2015. -
  4. [4][4] A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 2253-2261. Curran Associates, Inc., 2016. -
  5. [5][5] Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2 , NIPS'14, pages 2933-2941, Cambridge, MA, USA, 2014. MIT Press. -
  6. [6][6] A. G. de G. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations , 2018. -
  7. [7][7] A. G. de G. Matthews, J. Hron, R. E. Turner, and Z. Ghahramani. Sample-then-optimize posterior sampling for bayesian linear models. In NIPS workshop on Advances in Approximate Bayesian Inference , 2017. -
  8. [9][9] T. Gneiting. Strictly and non-strictly positive definite functions on spheres. Bernoulli , 19(4):1327-1349, 2013. -
  9. [10][10] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Networks. NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 , pages 2672-2680, jun 2014. -
  10. [11][11] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks , 2(5):359 - 366, 1989. -
  11. [13][13] J. H. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. ICLR , 2018. -
  12. [14][14] M. Leshno, V. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks , 6(6):861-867, 1993. -
  13. [15][15] S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018. -
  14. [16][16] R. M. Neal. Bayesian Learning for Neural Networks . Springer-Verlag New York, Inc., Secaucus, NJ, USA, 1996. -
  15. [17][17] R. Pascanu, Y. N. Dauphin, S. Ganguli, and Y. Bengio. On the saddle point problem for non-convex optimization. arXiv preprint , 2014. -
  16. [18][18] J. Pennington and Y. Bahri. Geometry of neural network loss surfaces via random matrix theory. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2798-2806, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. -
  17. [19][19] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20 , pages 1177-1184. Curran Associates, Inc., 2008. -
  18. [20][20] L. Sagun, U. Evci, V. U. G¨ uney, Y. Dauphin, and L. Bottou. Empirical analysis of the hessian of over-parametrized neural networks. CoRR , abs/1706.04454, 2017. - · arXiv
  19. [21][21] B. Sch¨ olkopf, A. Smola, and K.-R. M¨ uller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation , 10(5):1299-1319, 1998. -
  20. [22][22] J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis . Cambridge University Press, New York, NY, USA, 2004. -
  21. [23][23] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. ICLR 2017 proceedings , Feb 2017.

Cited by

Sorted by citations

Authors
Jacot, A., Gabriel, F., & Hongler, C.
Affiliations
  • École Polytechnique Fédérale de Lausanne (CH)
  • Imperial College London (GB)
ORCID
Hongler, Clément
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.07572
arXiv
1806.07572
OpenAlex
W2809090039
Cited by
1,562
References
0
Citations by year
2026: 7 · 2025: 131 · 2024: 201 · 2023: 188 · 2022: 164 · 2021: 414 · 2020: 333 · 2019: 113 · 2018: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Centres of Competence in Research SwissMAP
  • John S. Latsis Public Benefit Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
neural tangent kernel, infinite-width limit, generalization, gradient descent, kernel methods, Gaussian processes, positive definiteness, early stopping, least squares regression, wide neural networks, convergence analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review