Reconciling modern machine-learning practice and the classical bias–variance trade-off
Open accessAbstract
Significance: While breakthroughs in machine learning and artificial intelligence are changing society, our fundamental understanding has lagged behind. It is traditionally believed that fitting models to the training data exactly is to be avoided as it leads to poor performance on unseen data. However, powerful modern classifiers frequently have near-perfect fit in training, a disconnect that spurred recent intensive research and controversy on whether theory provides practical insights. In this work, we show how classical theory and modern practice can be reconciled within a single unified performance curve and propose a mechanism underlying its emergence. We believe this previously unknown pattern connecting the structure and performance of learning architectures will help shape design and understanding of learning algorithms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The implications of the variance-bias tradeoff can be more pronounced for sustainability solutions.
“While variance-bias tradeoff is an inseparable part of the ML debate (Belkin et al., 2019), implications can be more pronounced for sustainability solutions.”
From Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda (2020) · cites this paper as Belkin et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Self-citationThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138 · see citing works
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992 · see citing works
Gaussian Processes in Machine Learning
Rasmussen, C. E. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28650-9_4 · Cited by 5,727 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · see citing works
Random Features for Large-Scale Kernel Machines
Rahimi, A., & Recht, B. (2007) · Cited by 2,783 · see citing works
Boosting the margin: a new explanation for the effectiveness of voting methods
Bartlett, P. L., Freund, Y., Lee, W. S., & Schapire, R. E. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691352 · Cited by 2,541 · see citing works
Understanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400 · see citing works
Homo Heuristicus: Why Biased Minds Make Better Inferences
Gigerenzer, G., & Brighton, H. (2009) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2008.01006.x · Cited by 1,801 · see citing works
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,159 · see citing works
All of Nonparametric Statistics
Fotopoulos, S. (2007) · Technometrics · doi:10.1198/tech.2007.s454 · Cited by 1,141 · see citing works
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1987) · Information Processing Letters · doi:10.1016/0020-0190(87)90114-1 · Cited by 1,054 · see citing works
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · see citing works
Bühlmann, P., & Yu, B. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000125 · Cited by 838 · see citing works
Boosting the margin: A new explanation for the effectiveness of voting methods
Schapire, R. E., Freund, Y., Barlett, P., & Lee, W. S. (1997) · Cited by 711 · see citing works
All of Nonparametric Statistics
Wasserman, L. (2005) · Springer eBooks · doi:10.1007/0-387-30623-4 · Cited by 518 · see citing works
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks
Soltanolkotabi, M., Javanmard, A., & Lee, J. D. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2854560 · Cited by 333 · see citing works
High-dimensional dynamics of generalization error in neural networks
Advani, M., Saxe, A., & Sompolinsky, H. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.08.022 · Cited by 302 · see citing works
Explaining the Success of AdaBoost and Random Forests as Interpolating ClassifiersPreprint
Wyner, A. J., Olson, M., Bleich, J., & Mease, D. (2015) · arXiv (Cornell University) · doi:10.5555/3122009.3153004 · Cited by 244 · see citing works
Generalization Properties of Learning with Random FeaturesPreprint
Rudi, A., & Rosasco, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04474 · Cited by 183 · see citing works
Implicit Regularization in Matrix Factorization
Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., & Srebro, N. (2018) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2018.8503198 · Cited by 148 · see citing works
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
Belkin, M., Hsu, D., & Mitra, P. P. (2018) · Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) · doi:10.48550/arxiv.1806.05161 · Cited by 139 · see citing works
Self-citationA Modern Take on the Bias-Variance Tradeoff in Neural NetworksPreprint
Neal, B., Mittal, S., Baratin, A., Tantia, V., Scicluna, M., Lacoste-Julien, S., & Mitliagkas, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08591 · Cited by 138 · see citing works
Ma, S., Bassily, R., & Belkin, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06559 · Cited by 120 · see citing works
Self-citationDoes data interpolation contradict statistical optimality?Preprint
Belkin, M., Rakhlin, A., & Tsybakov, A. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09471 · Cited by 113 · see citing works
Self-citationTo understand deep learning we need to understand kernel learningPreprint
Belkin, M., Ma, S., & Mandal, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01396 · Cited by 99 · see citing works
Self-citationLearning Scalable Deep Kernels with Recurrent\nStructure
Al-Shedivat, M., Wilson, A. G., Saatchi, Y., Hu, Z., & Xing, E. P. (2017) · Europe PMC (PubMed Central) · Cited by 64 · see citing works
Li, Y., Ma, T., & Zhang, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09203 · Cited by 18 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [19][19] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256, 2010. -
- [22][22] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009. -
- [23][23] Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings , pages 331-339. Elsevier, 1995. -
- [24][24] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE , volume 86, pages 2278-2324, 1998. -
- [28][28] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop , volume 2011, page 4, 2011. -
- [29][29] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing , pages 1532-1543, 2014. -
- [37][37] Stefano Spigler, Mario Geiger, St´ ephane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665 , 2018. - · arXiv
- [40][40] Holger Wendland. Scattered Data Approximation . Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2004. doi: 10.1017/ CBO9780511617539. -
Cited by
Sorted by citations
Understanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400 · see citing works
Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda
Nishant, R., Kennedy, M., & Corbett, J. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102104 · Cited by 1,203 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Reproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“As a fundamental principle of model generalization theory, the bias–variance tradeoff indicates that a generalized model should maintain a balance between underfitting and overfitting [39, 124].”
BackgroundInfluentialThe Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538 · see citing works
Bibri, S. E., Krogstie, J., Kaboli, A., & Alahi, A. (2024) · Environmental Science and Ecotechnology · doi:10.1016/j.ese.2023.100330 · Cited by 536 · see citing works
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565 · see citing works
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397 · see citing works
Does learning require memorization? a short tale about a long tail
Feldman, V. (2020) · Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing · doi:10.1145/3357713.3384290 · Cited by 287 · see citing works
Integrative benchmarking to advance neurally mechanistic models of human intelligence
Schrimpf, M., Kubilius, J., Lee, M. J., Ratan Murty, N. A., Ajemian, R., & DiCarlo, J. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.040 · Cited by 233 · see citing works
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226 · see citing works
Deep learning for small and big data in psychiatry
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0767-z · Cited by 189 · see citing works
Cross-Validation Visualized: A Narrative Guide to Advanced Methods
Allgaier, J., & Pryss, R. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020065 · Cited by 176 · see citing works
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“17-20 Types and sources of bias One way to comprehend imaging AI bias is by examining sources of bias related to fundamental components of the AI life cycle: study design and dataset (formulating the research question, collection, annotation,”
Machine Learning Modeling for Accelerated Battery Materials Design in the Small Data Regime
Sendek, A. D., Ransom, B., Cubuk, E. D., Pellouchoud, L. A., Nanda, J., & Reed, E. J. (2022) · Advanced Energy Materials · doi:10.1002/aenm.202200553 · Cited by 100 · see citing works
Organizing memories for generalization in complementary learning systems
Sun, W., Advani, M., Spruston, N., Saxe, A., & Fitzgerald, J. E. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01382-9 · Cited by 103 · see citing works
On the Existence of Simpler Machine Learning Models
Semenova, L., Rudin, C., & Parr, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533232 · Cited by 90 · see citing works
Haines-Delmont, A., Chahal, G., Bruen, A. J., Wall, A., Khan, C. T., Sadashiv, R., & Fearnley, D. (2020) · JMIR mhealth and uhealth · doi:10.2196/15901 · Cited by 87 · see citing works
Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models
Rocks, J. W., & Mehta, P. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.013201 · Cited by 74 · see citing works
Must-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning
Koçak, B., Cuocolo, R., Santos, D. P. D., Stanzione, A., & Ugga, L. (2023) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2022.2022-11-51 · Cited by 64 · see citing works
Practically Adopting Human Activity Recognition
Xu, H., Zhou, P., Tan, R., & Li, M. (2023) · Proceedings of the 29th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3570361.3613299 · Cited by 55 · see citing works
Causal models for longitudinal and panel data: a survey
Arkhangelsky, D., & Imbens, G. (2024) · The Econometrics Journal · doi:10.1093/ectj/utae014 · Cited by 50 · see citing works
Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation
Chen, L., Chen, Y., Xi, J., & Le, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00248-y · Cited by 48 · see citing works
Chang, X., Li, Y., Oymak, S., & Thrampoulidis, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16859 · Cited by 24 · see citing works
NeuralDock: Rapid and Conformation-Agnostic Docking of Small Molecules
Sha, C. M., Wang, J., & Dokholyan, N. V. (2022) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2022.867241 · Cited by 9 · see citing works
A Hybrid Artificial Intelligence Framework for Melanoma Diagnosis Using Histopathological Images
Nogales, A., Garrido, M. C., Guitian, A., Rodriguez-Peralto, J.-L., Villanueva, C. P., Díaz-Prieto, D., & García-Tejedor, Á. J. (2025) · Technologies · doi:10.3390/technologies13080330 · Cited by 7 · see citing works
- Authors
- Belkin, M., Hsu, D., Ma, S., & Mandal, S.
- Affiliations
- The Ohio State University (US)
- Columbia University (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-07-24
- Volume
- 116
- Issue
- 32
- Pages
- 15849-15854
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6689936
- OpenAlex
- W2963518130
- Cited by
- 1,595
- References
- 35
- Citations by year
- 2026: 190 · 2025: 268 · 2024: 219 · 2023: 226 · 2022: 230 · 2021: 239 · 2020: 191 · 2019: 25 · 2018: 7
- FWCI
- 78.14
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (RI-1815697, CCF-1740833)
- Alfred P. Sloan Foundation (FG-2016-6295)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- double descent, bias-variance trade-off, overfitting, underfitting, artificial neural networks, interpolation, model capacity, generalization, performance curves, overparameterized models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
