Deep double descent: where bigger models and more data hurt*
Open accessAbstract
We show that a variety of modern deep learning tasks exhibit a ‘double-descent’ phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Over-parameterizing the networks and training them for longer typically resulted in improved performance.
“We typically noticed that over-parameterizing our networks and training them for longer resulted in improved performance (similar to the double-descent effect44).”
From Deep reinforcement learning can promote sustainable human behaviour in a common-pool resource problem (2025) · cites this paper as [44]
Penalisation is closely related to the bias-variance trade-off and aims to bring the model closer to the sweet spot of the bias-variance trade-off curve, where model performance in new data is maximised.
“Penalisation is closely related to the bias-variance trade-off depicted in figure 1, and is a method aiming to bring the model closer to the sweet spot of the bias-variance trade-off curve, where model performance in new data is maximised”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
(2017) · Cited by 11,625 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
Random Features for Large-Scale Kernel Machines
Rahimi, A., & Recht, B. (2007) · Cited by 2,783 · see citing works
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Benign overfitting in linear regression
Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907378117 · Cited by 554 · see citing works
WIT3: Web Inventory of Transcribed and Translated Talks
Cettolo, M., Girardi, C., & Federico, M. (2012) · Cited by 543 · see citing works
Surprises in high-dimensional ridgeless least squares interpolation
Hastie, T. J., Andrea Montanari, A., Rosset, S., & Tibshirani, R. J. (2022) · The Annals of Statistics · doi:10.1214/21-aos2133 · Cited by 510 · see citing works
High-dimensional dynamics of generalization error in neural networks
Advani, M., Saxe, A., & Sompolinsky, H. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.08.022 · Cited by 302 · see citing works
The generalization error of random features regression: Precise asymptotics and double descent curvePreprint
Mei, S., & Andrea Montanari, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05355 · Cited by 222 · see citing works
Scaling description of generalization with number of parameters in deep learning
Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d’Ascoli, S., Biroli, G., Hongler, C., & Wyart, M. (2020) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ab633c · Cited by 160 · see citing works
Harmless Interpolation of Noisy Data in Regression
Muthukumar, V., Vodrahalli, K., Subramanian, V., et al. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849614 · Cited by 135 · see citing works
Jamming transition as a paradigm to understand the loss landscape of deep neural networks
Geiger, M., Spigler, S., d’Ascoli, S., Sagun, L., Baity‐Jesi, M., Biroli, G., & Wyart, M. (2019) · Physical review. E · doi:10.1103/physreve.100.012115 · Cited by 127 · see citing works
SGD on Neural Networks Learns Functions of Increasing ComplexityPreprint
Nakkiran, P., Kaplun, G., Kalimeris, D., Yang, T., Edelman, B., Zhang, F., & Barak, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11604 · Cited by 94 · see citing works
Self-citationReconciling modern machine learning and the bias-variance trade-offPreprint
Belkin, M., Hsu, D., Ma, S., et al. (2018) · arXiv (Cornell University) · Cited by 84 · see citing works
A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
Bibas, K., Fogel, Y., & Feder, M. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849398 · Cited by 29 · see citing works
Graham, J. L. (1938) · The Psychological Monographs · doi:10.1037/h0093520 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Belkin M, Hsu D and Xu J 2019 Two models of double descent for weak features (arXiv:1903.07571) · arXiv
- Huang Y, Cheng Y, Chen D, Lee H, Ngiam J, Le Q V and Chen Z 2018 GPipe: efficient training of giant neural networks using pipeline parallelism (arXiv:1811.06965) · arXiv
- Krizhevsky A, Sutskever I and Hinton G E 2012 Imagenet classification with deep convolutional neural networks Advances in Neural Information Processing Systems pp 1097-105
- Mitra P P 2019 Understanding overfitting peaks in generalization error: analytical risk curves for l2 and l1 penalized interpolation (arXiv:1906.03667) · arXiv
- Opper M 1995 Statistical mechanics of learning: generalization The Handbook of Brain Theory and Neural Networks (Cambridge, MA: MIT Pres) pp 922-5
- Spigler S, Geiger M, d'Ascoli S, Sagun L, Biroli G and Wyart M 2018 A jamming transition from under-to over- parametrization affects loss landscape and generalization (arXiv:1810.09665) · arXiv
- Zhang C, Bengio S, Hardt M, Recht B and Vinyals O 2016 Understanding deep learning requires rethinking generalization ICLR (arXiv:1611.03530) · arXiv
Cited by
Sorted by citations
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538 · see citing works
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233 · see citing works
Explaining Neural Scaling Laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021) · arXiv · doi:10.48550/arxiv.2102.06701 · Cited by 182
Machine Learning Modeling for Accelerated Battery Materials Design in the Small Data Regime
Sendek, A. D., Ransom, B., Cubuk, E. D., Pellouchoud, L. A., Nanda, J., & Reed, E. J. (2022) · Advanced Energy Materials · doi:10.1002/aenm.202200553 · Cited by 100 · see citing works
Organizing memories for generalization in complementary learning systems
Sun, W., Advani, M., Spruston, N., Saxe, A., & Fitzgerald, J. E. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01382-9 · Cited by 103 · see citing works
On the Existence of Simpler Machine Learning Models
Semenova, L., Rudin, C., & Parr, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533232 · Cited by 90 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models
Rocks, J. W., & Mehta, P. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.013201 · Cited by 74 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47 · see citing works
Self-citationChang, X., Li, Y., Oymak, S., & Thrampoulidis, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16859 · Cited by 24 · see citing works
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
Koster, R., Pîslar, M., Tacchetti, A., Balaguer, J., Liu, L., Elie, R., Hauser, O. P., Tuyls, K., Botvinick, M., & Summerfield, C. (2025) · Nature Communications · doi:10.1038/s41467-025-58043-7 · Cited by 12 · see citing works
Gauge fixing for sequence-function relationships
Posfai, A., Zhou, J., McCandlish, D. M., & Kinney, J. B. (2024) · doi:10.1101/2024.05.12.593772 · Cited by 11 · see citing works
NeuralDock: Rapid and Conformation-Agnostic Docking of Small Molecules
Sha, C. M., Wang, J., & Dokholyan, N. V. (2022) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2022.867241 · Cited by 9 · see citing works
Reimagining cancer treatments in the era of generative AI
Derbal, Y. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251394631 · Cited by 4 · see citing works
- Authors
- Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I.
- Affiliations
- Harvard University (US)
- OpenAI (United States) (US)
- Corresponding author
- Nakkiran, Preetum
- Venue
- Journal of Statistical Mechanics: Theory and Experiment
- Publisher
- IOP Publishing
- Published
- 2021-12-01
- Volume
- 2021
- Issue
- 12
- Pages
- 124003
- ISSN
- 1742-5468 (ISSN-L)
- OpenAlex
- W2994081359
- Cited by
- 565
- References
- 24
- Citations by year
- 2026: 60 · 2025: 94 · 2024: 93 · 2023: 93 · 2022: 81 · 2021: 86 · 2020: 54 · 2019: 4
- FWCI
- 30.91
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Statistical Mechanics Theory and Experiment (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1409097, 1540428, 1715187, 1301976, 1565264, CAREER, 2014389, 1452961, 1618026)
- Simons Institute for the Theory of Computing, University of California Berkeley
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- double descent, model size, training data size, training epochs, test performance, overfitting, generalization, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
