Mathematical Models of Overparameterized Neural Networks
Open accessThe DOI matches its Crossref record; 122 of the 134 references it cites were found in a registry or an index.
Abstract
Deep learning has received considerable empirical success in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been a lack of theoretical understanding for tricks invented in the deep learning literature. Known by practitioners that overparameterized neural networks (NNs) are easy to learn, in the past few years, there have been important theoretical developments in the analysis of overparameterized NNs. In particular, it was shown that such systems behave like convex systems under various restricted settings, such as for two-layer NNs, and when learning is restricted locally in the so-called neural tangent kernel space around specialized initializations. This article discusses some of these recent signs of progress leading to a significantly better understanding of NNs. We will focus on the analysis of two-layer NNs and explain the key mathematical models, with their algorithmic implications. We will then discuss challenges in understanding deep NNs and some current research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures













Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides, channel perturbation in algorithm iterates can also serve as the mechanism to design saddle points escaping algorithms [94], thereby establishing global optimality for training the non-convex over-parameterized neural networks in high-dimensional statistical settings [180].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [180]
Experimental — accuracy not yet measuredIn the citing paper
“However, empirical risk minimization for training the models is usually nonconvex, which poses significant challenges to guarantee global optimality for the learning algorithms and models [180].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [180]
In the citing paper
“Besides, with high-dimensional datasets, the nonconvex loss functions of certain statistical learning models, including over-parameterized neural networks [180] and dictionary learning [256], can enjoy benign global geometric landscape such that all the local minima are global minima, and all the saddle points can be escaped efficiently using the algorithms including trust region method and pertur”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [180]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“3048020 deep NNs (DNNs) have largely improved the state-of-theart performances in numerous real applications, such as image classification [1], [2], speech recognition [3], and natural language processing [4].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“3048020 deep NNs (DNNs) have largely improved the state-of-theart performances in numerous real applications, such as image classification [1], [2], speech recognition [3], and natural language processing [4].”
BackgroundGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“com/exdb/mnist/ [42] to demonstrate various aspects of the different frameworks.”
Background(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 2×
“In real applications, initialization strategies, e.g., [74], [45] sample the NN weights from N (0 , O ( m )) , with variance approaching infinity as the number of hidden nodes m goes to ∞ .”
BackgroundMethodEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 1×
“It was observed, in practice, that it is beneficial to increase both depth and width simultaneously to balance the tradeoff [90].”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“3048020 deep NNs (DNNs) have largely improved the state-of-theart performances in numerous real applications, such as image classification [1], [2], speech recognition [3], and natural language processing [4].”
Background(2010) · Cited by 14,134
Research works printed but not linked: 3
- J. Sirignano and K. Spiliopoulos, "Mean field analysis of neural networks: A law of large numbers," SIAM J. Appl. Math., vol. 80, no. 2, pp. 725-752, Jan. 2020. · DOI
- J. Sirignano and K. Spiliopoulos, "Mean field analysis of neural networks: A central limit theorem," Stochastic Processes Their Appl., vol. 130, no. 3, pp. 1820-1852, Mar. 2020.
- D. Erhan, Y. Bengio, A. Courville, and P . Vincent, "Visualizing higher-layer features of a deep network," Univ. Montreal, vol. 1341, no. 3, p. 1, 2009.
Cited by
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“empirical risk minimization for training the models is usually nonconvex, which poses significant challenges to guarantee global optimality for the learning algorithms and models [180].”
BackgroundExploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training
Fang, C., He, H., Long, Q., & Su, W. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103091118 · Cited by 269
Self-citationA Layer-Wise Theoretical Framework for Deep Learning of Convolutional Neural Networks
Nguyen, H.-T., Li, S., & Cheah, C. C. (2022) · IEEE Access · doi:10.1109/access.2022.3147869 · Cited by 94
“For networks with standard structures, most studies have considered over-parameterization networks [17], [18] where the widths of the hidden layers are assumed to be very large so that the problem can be analyzed mathematically.”
BackgroundBayesian Invariant Risk Minimization
Yong, L., Dong, H., Wang, H., & Zhang, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01555 · Cited by 106
“Suppose the prediction model consists of a feature extractor and a classifier [16, 17].”
MethodSelf-citationMinimum Excess Risk in Bayesian Learning
Xu, A., & Raginsky, M. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3176056 · Cited by 52
“, when g : X×W → Y can be represented by over-parametrized neural networks [60].”
Background
Metadata
- Authors
- Fang, C., Dong, H., & Zhang, T.
- Affiliations
- University of Pennsylvania (US)
- Hong Kong University of Science and Technology (HK)
- Venue
- Proceedings of the IEEE
- Published
- 2021-05
- Volume
- 109
- Issue
- 5
- Pages
- 683-703
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W3118114660
- Cited by
- 26
- References
- 134
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 5 · 2023: 6 · 2022: 4 · 2021: 5
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- GRF (16201320)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- overparameterized neural networks, two-layer neural networks, neural tangent kernel, convexity, generalization, learning dynamics, deep learning theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mathematical Models of Overparameterized Neural Networks. https://app.cochat.ai/papers/10.1109/jproc.2020.3048020
- The paper
- Fang, C., Dong, H., & Zhang, T. (2021). Mathematical Models of Overparameterized Neural Networks. Proceedings of the IEEE. https://doi.org/10.1109/jproc.2020.3048020
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.