CoChat
Sign Up
Cited by 181Stochastic Gradient Optimization TechniquesNeural Networks and ApplicationsDomain Adaptation and Few-Shot Learning

Explaining neural scaling laws

Open access
Bahri, Yasaman, Dyer, Ethan, Kaplan, Jared, Lee, Jaehoon, Sharma, Utkarsh·2024·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.2311878121PMC PMC11228526
Open this paper in CoChat — free account

Abstract

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origin and relationships between scaling exponents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2311878121

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Theoretical accounts of power-law scaling are partial: they recover power-law forms and rationalize the dependence on data geometry, but they do not yet predict the precise coefficients of large transformer runs from first principles.

    “These theories are partial-they recover power-law forms and rationalize the dependence on data geometry, but they do not yet predict the precise coefficients of large transformer runs from first principles”

    From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Bahri et al., 2024

    Section 1.2 derives power-law exponents and relates them to data-manifold dimension. Section 4.1 says the theoretical results are asymptotic and identifies limits in applying them to finite models and datasets. The paper does not specifically assess whether theories can predict precise coefficients for large transformer runs from first principles. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Bahri et al. [2024] developed a taxonomy distinguishing variance-limited regimes from resolution-limited regimes, deriving exponents in each and connecting them to properties of the learned kernel.

    “Bahri et al. [2024] developed this into a taxonomy distinguishing variance-limited regimes (where one resource is abundant and error falls quickly) from resolution-limited regimes (where error is set by how finely the model resolves the data manifold), deriving exponents in each and connecting them to properties of the learned kernel.”

    From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Bahri et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. [1]J. Hestness et al., Deep learning scaling is predictable, empirically. arXiv [Preprint] (2017). 10.48550/arXiv.1712.00409 (Accessed 1 January 2021). · DOI · Paper page
  2. [3]J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, N. Shavit, “A constructive prediction of the generalization error across scales” in International Conference on Learning Representations (2020).
  3. [4]T. Henighan et al., Scaling laws for autoregressive generative modeling. arXiv [Preprint] (2020). 10.48550/arXiv.2010.14701 (Accessed 1 January 2021). · DOI · Paper page
  4. [5]J. S. Rosenfeld, J. Frankle, M. Carbin, N. Shavit, “On the predictability of pruning across scales” in International Conference on Machine Learning (PMLR, 2021), pp. 9075–9083.
  5. [6]S. Ahmad, G. Tesauro, “Scaling and generalization in neural networks: A case study” in Advances in Neural Information Processing Systems (1989), pp. 160–168.
  6. [7]D. Cohn, G. Tesauro, “Can neural networks do better than the Vapnik–Chervonenkis bounds?” in Advances in Neural Information Processing Systems (1991), pp. 911–917.
  7. [8]T. B. Brown et al., Language models are few-shot learners. arXiv [Preprint] (2020). 10.48550/arXiv.2010.14701 (Accessed 1 January 2021). · DOI · Paper page
  8. [9]J. Hoffmann, An empirical analysis of compute-optimal large language model training. Adv. Neural Inf. Process. Syst. 35, 30016– 30030 (2022).
  9. [11]D. Bisla, A. N. Saridena, A. Choromanska, A theoretical-empirical approach to estimating sample complexity of DNNs. arXiv [Preprint] (2021). 10.48550/arXiv.2105.01867 (Accessed 1 January 2021). · DOI
  10. [12]R. M. Neal, “Bayesian learning for neural networks,” PhD thesis, Department of Computer Science, University of Toronto, Toronto, ON, Canada (1994).
  11. [13]J. Lee et al., “Deep neural networks as Gaussian processes” in International Conference on Learning Representations (2018).
  12. [14]A. Matthews, J. Hron, M. Rowland, R. E. Turner, Z. Ghahramani, “Gaussian process behaviour in wide deep neural networks” in International Conference on Learning Representations (2018).
  13. [15]A. Jacot, F. Gabriel, C. Hongler, “Neural Tangent Kernel: Convergence and generalization in neural networks” in Advances in Neural Information Processing Systems (2018).
  14. [16]J. Lee et al., “Wide neural networks of any depth evolve as linear models under gradient descent” in Advances in Neural Information Processing Systems (2019).
  15. [17]E. Dyer, G. Gur-Ari, “Asymptotics of wide networks from Feynman diagrams” in International Conference on Learning Representations (2020).
  16. [18]S. Spigler, M. Geiger, M. Wyart, Asymptotic learning curves of kernel methods: Empirical data versus teacher-student paradigm. J. Stat. Mech.: Theory Exp. 2020, 124001 (2020).
  17. [19]B. Bordelon, A. Canatar, C. Pehlevan, “Spectrum dependent learning curves in kernel regression and wide neural networks” in International Conference on Machine Learning (PMLR, 2020), pp. 1024–1034.
  18. [21]I. Steinwart et al., “Optimal rates for regularized least squares regression” in COLT (2009), pp. 79–93.
  19. [23]M. Hutter, Learning curve theory. arXiv [Preprint] (2021). 10.48550/arXiv.2102.04074 (Accessed 10 February 2021). · DOI
  20. [24]H. Cui, B. Loureiro, F. Krzakala, L. Zdeborová, Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Adv. Neural Inf. Process. Syst. 34, 10131– 10143 (2021).
  21. [25]A. Maloney, D. A. Roberts, J. Sully, A solvable model of neural scaling laws. arXiv [Preprint] (2022). 10.48550/arXiv.2210.16859 (Accessed 1 November 2022). · DOI
  22. [26]A. Wei, W. Hu, J. Steinhardt, “More than a toy: Random matrix models predict how real-world neural representations generalize” in International Conference on Machine Learning (PMLR, 2022), pp. 23549–23588.
  23. [27]A. Rahimi, B. Recht, Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Adv. Neural Inf. Process. Syst. 21, 1313– 1320 (2008).
  24. [28]T. Hastie, A. Montanari, S. Rosset, R. J. Tibshirani, Surprises in high-dimensional ridgeless least squares interpolation. arXiv [Preprint] (2019). 10.48550/arXiv.1903.08560 (Accessed 1 January 2021). PMC9481183 36120512 · DOI
  25. [29]S. d’Ascoli, M. Refinetti, G. Biroli, F. Krzakala, “Double trouble in double descent: Bias and variance(s) in the lazy regime” in International Conference on Machine Learning (PMLR, 2020), pp. 2280–2290.
  26. [30]M. S. Advani, A. M. Saxe, High-dimensional dynamics of generalization error in neural networks. arXiv [Preprint] (2017). 10.48550/arXiv.1710.03667 (Accessed 1 January 2021). PMC7685244 33022471 · DOI
  27. [31]M. S. Advani, A. M. Saxe, H. Sompolinsky, High-dimensional dynamics of generalization error in neural networks. Neural Networks 132, 428– 446 (2020). 33022471 PMC7685244
  28. [32]S. Mei, A. Montanari, The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv [Preprint] (2019). 10.48550/arXiv.1908.05355 (Accessed 1 January 2021). · DOI
  29. [33]B. Adlam, J. Pennington, “The Neural Tangent Kernel in high dimensions: Triple descent and a multi-scale theory of generalization” in International Conference on Machine Learning (PMLR, 2020), pp. 74–84.
  30. [34]B. Adlam, J. Pennington, Understanding double descent requires a fine-grained bias-variance decomposition. Adv. Neural Inf. Process. Syst. 33, 11022– 11032 (2020).
  31. [35]A. Andreassen, E. Dyer, Asymptotics of wide convolutional neural networks. arxiv [Preprint] (2020). 10.48550/arXiv.2008.08675 (Accessed 1 January 2021). · DOI
  32. [36]M. Geiger, Scaling description of generalization with number of parameters in deep learning. J. Stat. Mech.: Theory Exp. 2020, 023401 (2020).
  33. [37]P. Nakkiran, B. Neyshabur, H. Sedghi, “The deep bootstrap framework: Good online learners are good offline generalizers” in International Conference on Learning Representations (2021).
  34. [42]M. L. Stein, Interpolation of Spatial Data: Some Theory for Kriging (Springer Science & Business Media, 1999).
  35. [43]P. J. Bickel et al., “Local polynomial regression on unknown manifolds” in Complex Datasets and Inverse Problems (Institute of Mathematical Statistics, 2007), pp. 177–186.
  36. [44]D. de Laat, “Approximating manifolds by meshes: Asymptotic bounds in higher codimension” Thesis, University of Groningen, Groningen, The Netherlands (2011).
  37. [45]S. Yaida, “Non-Gaussian processes and neural networks at finite widths” in Mathematical and Scientific Machine Learning Conference (2020).
  38. [46]B. Hanin, M. Nica, “Finite depth and width corrections to the neural tangent” in International Conference on Learning Representations (2020).
  39. [47]E. Levina, P. J. Bickel, “Maximum likelihood estimation of intrinsic dimension” in Advances in Neural Information Processing Systems (2005), pp. 777–784.
  40. [48]P. McCullagh, J. A. Nelder, Generalized Linear Models (CRC Press, 1989), vol. 37.
  41. [50]T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer Science & Business Media, 2009).
  42. [51]P. Nakkiran, More data can hurt for linear regression: Sample-wise double descent. arXiv [Preprint] (2019). 10.48550/arXiv.1912.07242 (Accessed 1 January 2021). · DOI
  43. [52]R. Novak et al., “Bayesian deep convolutional networks with many channels are Gaussian processes” in International Conference on Learning Representations (2019).
  44. [53]A. Garriga-Alonso, L. Aitchison, C. E. Rasmussen, “Deep convolutional networks as shallow Gaussian processes” in International Conference on Learning Representations (2019).
  45. [54]G. Yang, Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv [Preprint] (2019). 10.48550/arXiv.1902.04760 (Accessed 1 January 2021). · DOI
  46. [55]A. Lewkowycz, Y. Bahri, E. Dyer, J. Sohl-Dickstein, G. Gur-Ari, The large learning rate phase of deep learning: The catapult mechanism. arXiv [Preprint] (2020). 10.48550/arXiv.2003.02218 (Accessed 1 January 2021). · DOI
  47. [56]W. Huang, W. Du, R. Y. Da Xu, C. Liu, Implicit bias of deep linear networks in the large learning rate phase. arXiv [Preprint] (2020). 10.48550/arXiv.2011.12547 (Accessed 1 January 2021). · DOI
  48. [57]L. Chizat, E. Oyallon, F. Bach, “On lazy training in differentiable programming” in Advances in Neural Information Processing Systems (2019), pp. 2937–2947.
  49. [58]A. Loukas, “How close are the eigenvectors of the sample and actual covariance matrices?” in International Conference on Machine Learning (PMLR, 2017), pp. 2228–2237.
  50. [59]A. Canatar, B. Bordelon, C. Pehlevan, Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat. Commun. 12, 1– 12 (2021). 34006842 PMC8131612
  51. [61]P. Nakkiran, Y. Bansal, Distributional generalization: A new kind of generalization. arXiv [Preprint] (2020). 10.48550/arXiv.2009.08092 (Accessed 1 January 2021). · DOI
  52. [62]W. Grathwohl et al., “Your classifier is secretly an energy based model and you should treat it like one” in International Conference on Learning Representations (2020).

Cited by

Sorted by citations

Authors
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U.
Affiliations
  • Google (United States) (US)
  • Department of Physics and Astronomy, Johns Hopkins University, Baltimore, MD 21218
  • Johns Hopkins University (US)
Corresponding author
  • Bahri, Yasaman
  • Dyer, Ethan
  • Lee, Jaehoon
  • Sharma, Utkarsh
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2024-06-24
Volume
121
Issue
27
Pages
e2311878121-e2311878121
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.2311878121
PMC
PMC11228526
OpenAlex
W3133450183
Cited by
181
References
3
Citations by year
2026: 55 · 2025: 74 · 2024: 17 · 2023: 15 · 2022: 7 · 2021: 13
FWCI
21.55
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural scaling laws, power-law scaling, population decline, infinite-width limit, data manifold, random features, pre-trained models, scaling exponents, duality, model size, training data size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review