High-dimensional dynamics of generalization error in neural networks
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
We perform an analysis of the average generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant "high-dimensional" regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that standard application of theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 3×
“An empirical approach to test for memorization is to see if a deep neural network can fit a training set with randomly scrambled labels rather than the true labels (Zhang et al., 2017).”
BackgroundInfluentialUniversal approximation bounds for superpositions of a sigmoidal function
Barron, A. R. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.256500 · Cited by 2,804
Rademacher and Gaussian Complexities: Risk Bounds and Structural Results
Bartlett, P. L., & Mendelson, S. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_15 · Cited by 1,949
DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
Marčenko, V. A., & Pastur, L. A. (1967) · Mathematics of the USSR-Sbornik · doi:10.1070/sm1967v001n04abeh001994 · Cited by 1,713
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562
Spectral Analysis of Large Dimensional Random Matrices
Bai, Z., & Silverstein, J. W. (2010) · Springer Series in Statistics · doi:10.1007/978-1-4419-0661-8 · Cited by 1,544
Neural networks and principal component analysis: Learning from examples without local minima
Baldi, P., & Hornik, K. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90014-2 · Cited by 1,383 · cited 1×
“…a simple input-output map, deep linear networks retain several important features of nonlinear networks: most notably, the learning problem is nonconvex (P. Baldi and Hornik, 1989; Kawaguchi, 2016), yielding nonlinear dynamical trajectories (Fukumizu, 1998; Saxe, McClelland, and Ganguli, 2014).”
BackgroundA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122
On Early Stopping in Gradient Descent Learning
Yao, Y., Rosasco, L., & Caponnetto, A. (2007) · Constructive Approximation · doi:10.1007/s00365-006-0663-2 · Cited by 970
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 931
Self-citationExact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2014) · International Conference on Learning Representations · Cited by 761
Self-citationDeep learning without poor local minima
Kawaguchi, K. (2016) · arXiv · doi:10.48550/arxiv.1605.07110 · Cited by 592
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565
Benign overfitting in linear regression
Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907378117 · Cited by 547
Statistical mechanics of learning from examples
Seung, H. S., Sompolinsky, H., & Tishby, N. (1992) · Physical Review A · doi:10.1103/physreva.45.6056 · Cited by 532
Self-citationA mean field view of the landscape of two-layer neural networks
Mei, S., Montanari, A., & Nguyen, P.-M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806579115 · Cited by 500
The statistical mechanics of learning a rule
Watkin, T. L. H., Rau, A., & Biehl, M. (1993) · Reviews of Modern Physics · doi:10.1103/revmodphys.65.499 · Cited by 471
The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices
Benaych-Georges, F., & Nadakuditi, R. R. (2011) · Advances in Mathematics · doi:10.1016/j.aim.2011.02.007 · Cited by 441
Wide neural networks of any depth evolve as linear models under gradient descent *
Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2020) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/abc62b · Cited by 353
Asymptotic statistical theory of overtraining and cross-validation
Amari, S., Murata, N., Muller, K.-R., Finke, M., & Yang, H. H. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623200 · Cited by 351
Qualitatively characterizing neural network optimization problems
Goodfellow, I. J., Vinyals, O., & Saxe, A. M. (2014) · arXiv · doi:10.48550/arxiv.1412.6544 · Cited by 341
Self-citationThe singular values and vectors of low rank perturbations of large rectangular random matrices
Benaych-Georges, F., & Nadakuditi, R. R. (2012) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2012.04.019 · Cited by 301
Learning in linear neural networks: a survey
Baldi, P. F., & Hornik, K. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.392248 · Cited by 252
Statistical Mechanics of Deep Learning
Bahri, Y., Kadmon, J., Pennington, J., Schoenholz, S. S., Sohl-Dickstein, J., & Ganguli, S. (2020) · Annual Review of Condensed Matter Physics · doi:10.1146/annurev-conmatphys-031119-050745 · Cited by 236
A random matrix approach to neural networks
Louart, C., Liao, Z., & Couillet, R. (2018) · The Annals of Applied Probability · doi:10.1214/17-aap1328 · Cited by 232
Geometry of neural network loss surfaces via random matrix theory
Pennington, J., & Bahri, Y. (2017) · International Conference on Machine Learning · Cited by 183
Exact Solution for On-Line Learning in Multilayer Neural Networks
Saad, D., & Solla, S. A. (1995) · Physical Review Letters · doi:10.1103/physrevlett.74.4337 · Cited by 175
Eigenvalues of covariance matrices: Application to neural-network learning
Cun, Y. L., Kanter, I., & Solla, S. A. (1991) · Physical Review Letters · doi:10.1103/physrevlett.66.2396 · Cited by 146
Universality of covariance matrices
Pillai, N. S., & Yin, J. O. S. (2014) · The Annals of Applied Probability · doi:10.1214/13-aap939 · Cited by 145
Statistical mechanics of complex neural systems and high dimensional data
Advani, M., Lahiri, S., & Ganguli, S. (2013) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2013/03/p03014 · Cited by 122
Self-citationGeneralization in a linear perceptron in the presence of noise
Krogh, A., & Hertz, J. A. (1992) · Journal of Physics A: Mathematical and General · doi:10.1088/0305-4470/25/5/020 · Cited by 116
Dynamics of Learning Near Singularities in Layered Networks
Wei, H., Zhang, J., Cousseau, F., Ozeki, T., & Амари, Ш.-И. (2007) · Neural Computation · doi:10.1162/neco.2007.12-06-414 · Cited by 66
Statistical Mechanics of Optimal Convex Inference in High Dimensions
Advani, M., & Ganguli, S. (2016) · Physical Review X · doi:10.1103/physrevx.6.031034 · Cited by 46
Self-citationTemporal Evolution of Generalization during Learning in Linear Networks
Baldi, P., & Chauvin, Y. (1991) · Neural Computation · doi:10.1162/neco.1991.3.4.589 · Cited by 41
Hoyle, D. C., & Rattray, M. (2007) · Physical Review E · doi:10.1103/physreve.75.016101 · Cited by 36
Learning and generalization in a linear perceptron stochastically trained with noisy data
Dunmur, A. P., & Wallace, D. J. (1993) · Journal of Physics A Mathematical and General · doi:10.1088/0305-4470/26/21/016 · Cited by 23
An equivalence between high dimensional Bayes optimal inference and M-estimationPreprint
Advani, M., & Ganguli, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07060 · Cited by 9
Self-citation
Cited by
Machine learning and the physical sciences
Carleo, G., Cirac, I., Cranmer, K., Daudet, L., Schuld, M., Tishby, N., Vogt-Maranto, L., & Zdeborová, L. (2019) · Reviews of Modern Physics · doi:10.1103/revmodphys.91.045002 · Cited by 2,417
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
“We note that this transition from under-to over-parameterized regimes for neural networks was also previously observed by [7, 1, 27, 37].”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as 1 /D .”
BackgroundA deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027
Self-citationDeep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565
Metadata
- Authors
- Advani, M. S., Saxe, A. M., & Sompolinsky, H.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-12
- Volume
- 132
- Pages
- 428-446
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2763894180
- Cited by
- 289
- References
- 43
- FWCI
- 16.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.