Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models
Open accessAbstract
the bias and variance in contrast with the classical bias-variance trade-off. We also show that in contrast with classical intuition, over-parameterized models can overfit even in the absence of noise and exhibit bias even if the student and teacher models match. We synthesize these results to construct a holistic understanding of generalization error and the bias-variance trade-off in over-parameterized models and relate our results to random matrix theory.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The technical evolution of ML models, especially large neural networks, has begun to challenge traditional validation error minimization as the ideal strategy to optimize the bias-variance tradeoff during model training.
“In recent years, the technical evolution of ML models, and especially the rise of large neural network architectures, has begun to challenge the traditional approach of validation (or cross-validation) error minimization as the ideal strategy to optimize the bias-variance tradeoff during model training.17, 18, 19, 20”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · see citing works
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
Marčenko, V., & Pastur, L. (1967) · Mathematics of the USSR-Sbornik · doi:10.1070/sm1967v001n04abeh001994 · Cited by 2,805 · see citing works
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,782 · see citing works
Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,450 · see citing works
International Conference on Learning Representations (ICLR)
Abbas, M., Kivinen, J., & Raiko, T. (2016) · Cited by 1,632 · see citing works
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · see citing works
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562 · see citing works
A high-bias, low-variance introduction to Machine Learning for physicists
Mehta, P., Bukov, M., Wang, C.-H., Day, A. G. R., Richardson, C., Fisher, C. K., & Schwab, D. J. (2019) · Physics Reports · doi:10.1016/j.physrep.2019.03.001 · Cited by 1,104 · see citing works
Self-citationAn Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · see citing works
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · see citing works
Wide neural networks of any depth evolve as linear models under gradient descent *
Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl‐Dickstein, J., & Pennington, J. (2020) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/abc62b · Cited by 617 · see citing works
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565 · see citing works
Benign overfitting in linear regression
Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907378117 · Cited by 554 · see citing works
Empirical Analysis of the Hessian of Over-Parametrized Neural NetworksPreprint
Sagun, L., Evci, U., Güney, V., Dauphin, Y., & Bottou, L. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1706.04454 · Cited by 552 · see citing works
Surprises in high-dimensional ridgeless least squares interpolation
Hastie, T. J., Andrea Montanari, A., Rosset, S., & Tibshirani, R. J. (2022) · The Annals of Statistics · doi:10.1214/21-aos2133 · Cited by 510 · see citing works
Statistical Mechanics of Learning
Engel, A. K., & den Broeck, C. V. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9781139164542 · Cited by 490 · see citing works
The Cavity Method at Zero Temperature
Mézard, M., & Parisi, G. (2003) · Journal of Statistical Physics · doi:10.1023/a:1022221005097 · Cited by 371 · see citing works
High-dimensional dynamics of generalization error in neural networks
Advani, M., Saxe, A., & Sompolinsky, H. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.08.022 · Cited by 302 · see citing works
Entropy-SGD: biasing gradient descent into wide valleys*
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., & Zecchina, R. (2019) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ab39d9 · Cited by 294 · see citing works
Statistical Mechanics of Deep Learning
Bahri, Y., Kadmon, J., Pennington, J., Schoenholz, S. S., Sohl‐Dickstein, J., & Ganguli, S. (2019) · Annual Review of Condensed Matter Physics · doi:10.1146/annurev-conmatphys-031119-050745 · Cited by 237 · see citing works
Optimal errors and phase transitions in high-dimensional generalized linear models
Barbier, J., Krząkała, F., Macris, N., Miolane, L., & Zdeborová, L. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802705116 · Cited by 209 · see citing works
Mei, S., & Andrea Montanari, A. (2021) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.22008 · Cited by 203 · see citing works
Just interpolate: Kernel “Ridgeless” regression can generalizePreprint
Liang, T., & Rakhlin, A. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 190 · see citing works
Scaling description of generalization with number of parameters in deep learning
Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d’Ascoli, S., Biroli, G., Hongler, C., & Wyart, M. (2020) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ab633c · Cited by 160 · see citing works
A jamming transition from under- to over-parametrization affects generalization in deep learning
Spigler, S., Geiger, M., d’Ascoli, S., Sagun, L., Biroli, G., & Wyart, M. (2019) · Journal of Physics A Mathematical and Theoretical · doi:10.1088/1751-8121/ab4c8b · Cited by 140 · see citing works
2003 IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP '03
(2003) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2003.1202277 · Cited by 136 · see citing works
Harmless Interpolation of Noisy Data in Regression
Muthukumar, V., Vodrahalli, K., Subramanian, V., et al. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849614 · Cited by 135 · see citing works
Franz, S., Giorgio Parisi, A., Sevelev, M., Urbani, P., & Zamponi, F. (2017) · SciPost Physics · doi:10.21468/scipostphys.2.3.019 · Cited by 132 · see citing works
Two Models of Double Descent for Weak Features
Belkin, M., Hsu, D., & Xu, J. (2020) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m1336072 · Cited by 130 · see citing works
Jamming transition as a paradigm to understand the loss landscape of deep neural networks
Geiger, M., Spigler, S., d’Ascoli, S., Sagun, L., Baity‐Jesi, M., Biroli, G., & Wyart, M. (2019) · Physical review. E · doi:10.1103/physreve.100.012115 · Cited by 127 · see citing works
Nonlinear random matrix theory for deep learning*
Pennington, J., & Worah, P. (2019) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ab3bc3 · Cited by 112 · see citing works
Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model
Goldt, S., Mézard, M., Krząkała, F., et al. (2020) · Physical Review X · doi:10.1103/physrevx.10.041044 · Cited by 112 · see citing works
Mean-field message-passing equations in the Hopfield model and its generalizations
Mézard, M. (2017) · Physical review. E · doi:10.1103/physreve.95.022117 · Cited by 109 · see citing works
Generalisation error in learning with random features and the hidden manifold model*
Gerace, F., Loureiro, B., Krząkała, F., Mézard, M., & Zdeborová, L. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac3ae6 · Cited by 106 · see citing works
Rethinking Bias-Variance Trade-off for Generalization of Neural NetworksPreprint
Yang, Z., Yu, Y., You, C., Steinhardt, J., & Ma, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11328 · Cited by 102 · see citing works
Just interpolate: Kernel “Ridgeless” regression can generalize
Liang, T., & Rakhlin, A. (2020) · The Annals of Statistics · doi:10.1214/19-aos1849 · Cited by 98 · see citing works
Disentangling feature and lazy training in deep neural networks
Geiger, M., Spigler, S., Jacot, A., & Wyart, M. (2020) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/abc4de · Cited by 80 · see citing works
Statistical physics of community ecology: a cavity solution to MacArthur’s consumer resource model
Advani, M., Bunin, G., & Mehta, P. (2018) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/aab04e · Cited by 73 · see citing works
Self-citationSurprises in High-Dimensional Ridgeless Least Squares InterpolationPreprint
Hastie, T., Andrea Montanari, A., Rosset, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.08560 · Cited by 63 · see citing works
2019 IEEE International Symposium on Information Theory (ISIT)
(2019) · doi:10.1109/isit43479.2019 · Cited by 52 · see citing works
An analytic theory of generalization dynamics and transfer learning in deep linear networksPreprint
Lampinen, A. K., & Ganguli, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10374 · Cited by 50 · see citing works
The cavity method at zero temperaturePreprint
Mézard, M., & Parisi, G. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cond-mat/0207121 · Cited by 46 · see citing works
Analytic Study of Double Descent in Binary Classification: The Impact of Loss
Kini, G. R., & Thrampoulidis, C. (2020) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit44484.2020.9174344 · Cited by 41 · see citing works
Liao, Z., Couillet, R., & Mahoney, M. W. (2020) · arXiv (Cornell University) · doi:10.1088/1742-5468/ac3a77 · Cited by 41 · see citing works
Exact expressions for double descent and implicit regularization via surrogate random designPreprint
Dereziński, M., Liang, F., & Mahoney, M. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04533 · Cited by 38 · see citing works
More Data Can Hurt for Linear Regression: Sample-wise Double DescentPreprint
Nakkiran, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07242 · Cited by 37 · see citing works
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint
Ba, J. L., Erdogdu, M. A., Suzuki, T., et al. (2020) · International Conference on Learning Representations · Cited by 31 · see citing works
A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
Bibas, K., Fogel, Y., & Feder, M. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849398 · Cited by 29 · see citing works
Mehta, P., Cui, W., Wang, C.-H., & Marsland, R. (2019) · Physical review. E · doi:10.1103/physreve.99.052111 · Cited by 22 · see citing works
Self-citationUnderstanding Double Descent Requires a Fine-Grained Bias-Variance DecompositionPreprint
Adlam, B., & Pennington, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03321 · Cited by 22 · see citing works
On the number of variables to use in principal component regressionPreprint
Xu, J., & Hsu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01139 · Cited by 21 · see citing works
Mailman, M., & Chakraborty, B. (2011) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2011/07/l07002 · Cited by 21 · see citing works
A Precise Performance Analysis of Learning with Random FeaturesPreprint
Dhifallah, O., & Lu, Y. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11904 · Cited by 18 · see citing works
What causes the test error? Going beyond bias-variance via ANOVAPreprint
Lin, L., & Dobriban, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05170 · Cited by 17 · see citing works
A Model of Double Descent for High-Dimensional Logistic Regression
Deng, Z., Kammoun, A., & Thrampoulidis, C. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053524 · Cited by 14 · see citing works
Triple descent and the two kinds of overfitting: where and why do they appear?*
d’Ascoli, S., Sagun, L., & Biroli, G. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac3909 · Cited by 14 · see citing works
Benign Overfitting and Noisy Features
Li, Z., Su, W., & Sejdinović, D. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2093206 · Cited by 14 · see citing works
Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
Baldassi, C., Malatesta, E. M., Negri, M., et al. (2020) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/abcd31 · Cited by 10 · see citing works
Benign Overfitting and Noisy FeaturesPreprint
Zhu, L., Su, W., & Sejdinović, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02901 · Cited by 5 · see citing works
Cui, W., Rocks, J. W., & Mehta, P. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00663 · Cited by 5 · see citing works
Self-citationThe cavity method for analysis of large-scale penalized regressionPreprint
Ramezanali, M., Mitra, P. P., & Sengupta, A. M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.03194 · Cited by 3 · see citing works
Ramezanali, M., Mitra, P. P., & Sengupta, A. M. (2019) · Journal of Statistical Physics · doi:10.1007/s10955-019-02292-6 · Cited by 2 · see citing works
The Geometry of Over-parameterized Regression and Adversarial PerturbationsPreprint
Rocks, J. W., & Mehta, P. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14108 · Cited by 2 · see citing works
Self-citationEntropic gradient descent algorithms and wide flat minima*
Pittorino, F., Lucibello, C., Feinauer, C., et al. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac3ae8 · Cited by 2 · see citing works
A brief prehistory of double descent
Loog, M., Viering, T. J., Mey, A., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2001875117 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [26]D’Ascoli Stéphane, Refinetti Maria, Biroli Giulio, and Krzakala Florent, “ Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime,” Proceedings of the 37th International Conference on Machine Learning (ICML) PMLR, 119, 2280– 2290 (2020).
- [31]Jacot Arthur, Şimşek Berfin, Spadaro Francesco, Hongler Clément, and Gabriel Franck, “ Implicit regularization of random feature models,” Proceedings of the 37th International Conference on Machine Learning (ICML) PMLR, 119, 4631– 4640 (2020).
- [36]Liang Tengyuan, Rakhlin Alexander, and Zhai Xiyu, “ On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels,” Proceedings of Thirty Third Conference on Learning Theory PMLR, 125, 2683– 2711 (2020).
- [39]Mitra Partha P, “ Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolation,” (2019), arXiv:1906.03667. · arXiv
- [62]Yaida Sho, “ Non-Gaussian processes and neural networks at finite widths,” Proceedings of The First Mathematical and Scientific Machine Learning Conference PMLR, 107, 165– 192 (2020).
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
- Authors
- Rocks, J. W., & Mehta, P.
- Affiliations
- Boston University (US)
- Venue
- Physical Review Research
- Publisher
- American Physical Society (APS)
- Published
- 2022-03-15
- Volume
- 4
- Issue
- 1
- ISSN
- 2643-1564 (ISSN-L)
- PMC
- PMC9879296
- OpenAlex
- W3096727283
- Cited by
- 74
- References
- 68
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 24 · 2023: 7 · 2022: 9 · 2021: 2 · 2020: 1
- FWCI
- 4.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Physical Review Research (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Institutes of Health
- National Institute of General Medical Sciences (1R35GM119461)
- Boston University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- memorization, overfitting, bias-variance trade-off, overparameterized models, generalization error, interpolation, random matrix theory, teacher-student models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
