CoChat

Mathematical Models of Overparameterized Neural Networks

Open access
Fang, Cong, Dong, Hanze, Zhang, Tong · 2021 · Proceedings of the IEEE

The DOI matches its Crossref record; 122 of the 134 references it cites were found in a registry or an index.

Abstract

Deep learning has received considerable empirical success in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been a lack of theoretical understanding for tricks invented in the deep learning literature. Known by practitioners that overparameterized neural networks (NNs) are easy to learn, in the past few years, there have been important theoretical developments in the analysis of overparameterized NNs. In particular, it was shown that such systems behave like convex systems under various restricted settings, such as for two-layer NNs, and when learning is restricted locally in the so-called neural tangent kernel space around specialized initializations. This article discusses some of these recent signs of progress leading to a significantly better understanding of NNs. We will focus on the analysis of two-layer NNs and explain the key mathematical models, with their algorithmic implications. We will then discuss challenges in understanding deep NNs and some current research directions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/jproc.2020.3048020

Cites

122 of 122 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Solution neighborhood size versus NN width m.
Fig. 1 Solution neighborhood size versus NN width m.CC-BY · From the PDF
Fig. 2 Impact of α for NN optimization ( η u /η θ BP 1 , m BP 10 4 ).
Fig. 2 Impact of α for NN optimization ( η u /η θ BP 1 , m BP 10 4 ).CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Fig. 3 Impact of m for NN optimization ( α BP √ m). (a) η u /η θ BP m /α . (b) ηu/η θ BP 1. Fig 3
Fig. 3 Impact of m for NN optimization ( α BP √ m). (a) η u /η θ BP m /α . (b) ηu/η θ BP 1. Fig 3CC-BY · From the PDF
Fig. 4 Linearization of NN (first row: MNIST; second row: synthetic data).
Fig. 4 Linearization of NN (first row: MNIST; second row: synthetic data).CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Fig. 6 NN optimization from an MF perspective. (a) Change of loss. (b) Initial DI θ . (c) Optimized θ . (d) Normalized histogram of u. (e) Optimized θ with large u. (f) Optimized with noisy gradient. (g) Normalized histogram of u (noisy). (h) Optimized with large u (noisy).
Fig. 6 NN optimization from an MF perspective. (a) Change of loss. (b) Initial DI θ . (c) Optimized θ . (d) Normalized histogram of u. (e) Optimized θ with large u. (f) Optimized with noisy gradient. (g) Normalized histogram of u (noisy). (h) Optimized with large u (noisy).CC-BY · From the PDF
Fig. 7 Scaling factor α controls NN behavior: NTK versus MF .
Fig. 7 Scaling factor α controls NN behavior: NTK versus MF .CC-BY · From the PDF
Fig. 8 Visualization of weights on the MNIST data set (reshaped to 28 × 28). (a) Random. (b) Optimized. (c) Generated.
Fig. 8 Visualization of weights on the MNIST data set (reshaped to 28 × 28). (a) Random. (b) Optimized. (c) Generated.CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Fig. 10 Tangent space comparison (m BP 10 3 ).
Fig. 10 Tangent space comparison (m BP 10 3 ).CC-BY · From the PDF
Image 13
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. J. Sirignano and K. Spiliopoulos, "Mean field analysis of neural networks: A law of large numbers," SIAM J. Appl. Math., vol. 80, no. 2, pp. 725-752, Jan. 2020. · DOI
  2. J. Sirignano and K. Spiliopoulos, "Mean field analysis of neural networks: A central limit theorem," Stochastic Processes Their Appl., vol. 130, no. 3, pp. 1820-1852, Mar. 2020.
  3. D. Erhan, Y. Bengio, A. Courville, and P . Vincent, "Visualizing higher-layer features of a deep network," Univ. Montreal, vol. 1341, no. 3, p. 1, 2009.

Cited by

See all 26

Metadata

Authors
Fang, C., Dong, H., & Zhang, T.
Affiliations
  • University of Pennsylvania (US)
  • Hong Kong University of Science and Technology (HK)
Venue
Proceedings of the IEEE
Published
2021-05
Volume
109
Issue
5
Pages
683-703
ISSN
  • 0018-9219 (ISSN-L)
  • 1558-2256
OpenAlex
W3118114660
Cited by
26
References
134
Citations by year
2026: 2 · 2025: 4 · 2024: 5 · 2023: 6 · 2022: 4 · 2021: 5
FWCI
1.77
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the IEEE (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
GRF (16201320)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
overparameterized neural networks, two-layer neural networks, neural tangent kernel, convexity, generalization, learning dynamics, deep learning theory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.