Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Deep Double Descent: Where Bigger Models and More Data Hurt

Nakkiran, Preetum, Kaplun, Gal, Bansal, Yamini, Yang, Tristan, Barak, Boaz, Sutskever, Ilya·2019·arXiv
DOI 10.48550/arxiv.1912.02292arXiv 1912.02292
Open this paper in CoChat — free account

Abstract

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1912.02292

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 29

  1. Advani M S and Saxe A M 2017 High-dimensional dynamics of generalization error in neural networks (arXiv:1710.03667) · arXiv
  2. Bartlett P L, Long P M, Lugosi G and Alexander T 2019 Benign overfitting in linear regression (arXiv:1906.11300) · arXiv
  3. Belkin M, Hsu D, Ma S and Mandal S 2018 Reconciling modern machine learning and the bias-variance trade-off (arXiv:1812.11118) · arXiv
  4. Belkin M, Hsu D and Xu J 2019 Two models of double descent for weak features (arXiv:1903.07571) · arXiv
  5. Bibas K, Fogel Y and Feder M 2019 A new look at an old problem: a universal learning approach to linear regression (arXiv:1905.04708) · arXiv
  6. Cettolo M, Girardi C and Federico M 2012 Wit 3 : web inventory of transcribed and translated talks Proc. 16th Conf. European Association for Machine Translation (EAMT) (Trento, Italy) pp 261-8
  7. Geiger M, Arthur J, Spigler S, Gabriel F, Sagun L, d'Ascoli S, Biroli G, Hongler C and Wyart M 2019a Scaling description of generalization with number of parameters in deep learning (arXiv:1901.01608) · arXiv
  8. Geiger M, Spigler S, d'Ascoli S, Sagun L, Baity-Jesi M, Biroli G and Wyart M 2019b Jamming transition as a paradigm to understand the loss landscape of deep neural networks Phys. Rev. E 100 012115 · DOI · Paper page
  9. Goodfellow I J, Shlens J and Szegedy C 2014 Explaining and harnessing adversarial examples (arXiv:1412.6572) · arXiv
  10. Hastie T, Tibshirani R, Friedman J and Franklin J 2005 The Elements of Statistical Learning (Berlin: Springer) (https://doi.org/10.1007/b94608) · DOI · Paper page
  11. Hastie T, Montanari A, Rosset S and RyanTibshirani J 2019 Surprises in high-dimensional ridgeless least squares interpolation (arXiv:1903.08560) · arXiv
  12. K, Zhang X, Ren S and Sun J 2016 Identity mappings in deep residual networks European Conf. Computer Vision (Springer) pp 630-45
  13. Huang Y, Cheng Y, Chen D, Lee H, Ngiam J, Le Q V and Chen Z 2018 GPipe: efficient training of giant neural networks using pipeline parallelism (arXiv:1811.06965) · arXiv
  14. Krizhevsky A 2009 Learning multiple layers of features from tiny images Technical Report University of Toronto (https://doi.org/10.1.1.222.9220)
  15. Krizhevsky A, Sutskever I and Hinton G E 2012 Imagenet classification with deep convolutional neural networks Advances in Neural Information Processing Systems pp 1097-105
  16. Madry A, Makelov A, Schmidt L, Tsipras D and Vladu A 2017 Towards deep learning models resistant to adversarial attacks (arXiv:1706.06083) · arXiv
  17. Mei S and Montanari A 2019 The generalization error of random features regression: precise asymptotics and double descent curve (arXiv:1908.05355) · arXiv
  18. Mitra P P 2019 Understanding overfitting peaks in generalization error: analytical risk curves for l2 and l1 penalized interpolation (arXiv:1906.03667) · arXiv
  19. Muthukumar V, Vodrahalli K and Sahai A 2019 Harmless interpolation of noisy data in regression (arXiv:1903.09139) · arXiv
  20. Nakkiran P, Kaplun G, Kalimeris D, Yang T, Edelman B L, Zhang F and Barak B 2019 SGD on neural networks learns functions of increasing complexity (arXiv:1905.11604) · arXiv
  21. Opper M 1995 Statistical mechanics of learning: generalization The Handbook of Brain Theory and Neural Networks (Cambridge, MA: MIT Pres) pp 922-5
  22. Ott M, Edunov S, Baevski A, Fan A, Gross S, Ng N, Grangier D and Auli M 2019 fairseq: a fast, extensible toolkit for sequence modeling Proc. NAACL-HLT 2019: Demonstrations
  23. Paszke A et al 2017 Automatic differentiation in PyTorch NeurIPS Autodiff Workshop
  24. Rahimi A and Recht B 2008 Random features for large-scale kernel machines Advances in Neural Information Processing Systems pp 1177-84
  25. Sennrich R, Haddow B and Birch A 2015 Neural machine translation of rare words with subword units (arXiv:1508.07909) · arXiv
  26. Spigler S, Geiger M, d'Ascoli S, Sagun L, Biroli G and Wyart M 2018 A jamming transition from under-to over- parametrization affects loss landscape and generalization (arXiv:1810.09665) · arXiv
  27. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vincent V and Rabinovich A 2015 Going deeper with convolutions Computer Vision and Pattern Recognition (CVPR) (arXiv:1409.4842) · arXiv
  28. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L and Polosukhin I 2017 Attention is all you need (arXiv:1706.03762) · arXiv
  29. Zhang C, Bengio S, Hardt M, Recht B and Vinyals O 2016 Understanding deep learning requires rethinking generalization ICLR (arXiv:1611.03530) · arXiv

Cited by

Metadata

Authors
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I.
Venue
arXiv
Publisher
arXiv
Published
2019
DOI
10.48550/arxiv.1912.02292
arXiv
1912.02292
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.