CoChat
Sign Up
PreprintCited by 187Stochastic Gradient Optimization TechniquesNeural Networks and ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 85 of the 91 references it cites were found in a registry or an index.

Explaining Neural Scaling Laws

Open access
Bahri, Yasaman, Dyer, Ethan, Kaplan, Jared, Lee, Jaehoon, Sharma, Utkarsh·2021·arXiv
DOI 10.48550/arxiv.2102.06701arXiv 2102.06701PMC PMC11228526
Open this paper in CoChat — free account

Abstract

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origins of and relationships between scaling exponents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.06701

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. (A) Four scaling regimes. Here, we exhibit the four regimes we focus on in this work. (Top-Left and Bottom-Right) Variance-limited scaling of underparameterized models with dataset size and overparameterized models with number of parameters (width) exhibit universal scaling (αD=αW=1) independent of the architecture or underlying dataset. (Top-Right and Bottom-Left) Resolution-limited overparameterized models with dataset or underparameterized models with model size exhibit scaling with exponents that depend on the details of the data distribution. These four regimes are also found in random feature (Fig. 2A) and pretrained models (SI Appendix). (B) Resolution-limited models interpolate the data manifold. Linear interpolation between two training points in a four-dimensional input space (Top). We show a teacher model and four student models, each trained on different-sized datasets. In all cases, teacher and student approximately agree on the training endpoints, but as the training set size increases they increasingly match everywhere. (Bottom) We show 4/αD versus the data manifold dimension (input dimension for teacher-student models, intrinsic dimension for standard datasets). We find that the teacher-student models follow the 4/αD (dark dashed line), while the relationship for a four-layer convolutional neural network (solid) and Wide ResNet architecture (hollow) on standard datasets is less clear.
Fig. 1. (A) Four scaling regimes. Here, we exhibit the four regimes we focus on in this work. (Top-Left and Bottom-Right) Variance-limited scaling of underparameterized models with dataset size and overparameterized models with number of parameters (width) exhibit universal scaling (αD=αW=1) independent of the architecture or underlying dataset. (Top-Right and Bottom-Left) Resolution-limited overparameterized models with dataset or underparameterized models with model size exhibit scaling with exponents that depend on the details of the data distribution. These four regimes are also found in random feature (Fig. 2A) and pretrained models (SI Appendix). (B) Resolution-limited models interpolate the data manifold. Linear interpolation between two training points in a four-dimensional input space (Top). We show a teacher model and four student models, each trained on different-sized datasets. In all cases, teacher and student approximately agree on the training endpoints, but as the training set size increases they increasingly match everywhere. (Bottom) We show 4/αD versus the data manifold dimension (input dimension for teacher-student models, intrinsic dimension for standard datasets). We find that the teacher-student models follow the 4/αD (dark dashed line), while the relationship for a four-layer convolutional neural network (solid) and Wide ResNet architecture (hollow) on standard datasets is less clear.CC BY-NC-ND · PubMed Central
Fig. 2. (A) Random feature models exhibit all four scaling regimes. Here, we consider linear teacher-student models with random features trained with MSE loss to convergence. We see both variance-limited scaling (Top-Left and Bottom-Right) and resolution-limited scaling (Top-Right and Bottom-Left). Data are varied by downsampling MNIST by the specified pool size. (B) Duality and spectra in random feature models. Here, we show the relation between the decay of the kernel spectra, αK, and the scaling of the loss with number of data points, αD, and with number of parameters, αP. (Top) The spectra of kernels derived from random fully-connected deep neural networks on pooled MNIST (Bottom) appear well described by a power-law decay. The theoretical relation αD=αP=αK is given by the dashed black line.
Fig. 2. (A) Random feature models exhibit all four scaling regimes. Here, we consider linear teacher-student models with random features trained with MSE loss to convergence. We see both variance-limited scaling (Top-Left and Bottom-Right) and resolution-limited scaling (Top-Right and Bottom-Left). Data are varied by downsampling MNIST by the specified pool size. (B) Duality and spectra in random feature models. Here, we show the relation between the decay of the kernel spectra, αK, and the scaling of the loss with number of data points, αD, and with number of parameters, αP. (Top) The spectra of kernels derived from random fully-connected deep neural networks on pooled MNIST (Bottom) appear well described by a power-law decay. The theoretical relation αD=αP=αK is given by the dashed black line.CC BY-NC-ND · PubMed Central
Fig. 3. Effect of data distribution on scaling exponents. For CIFAR-100 superclassed to N classes (Left), we find that the number of target classes does not have a visible effect on the scaling exponent. (Right) For CIFAR-10 with the addition of Gaussian noise to inputs, we find the strength of the noise has a strong effect on performance scaling with dataset size. All models are WRN-28-10.
Fig. 3. Effect of data distribution on scaling exponents. For CIFAR-100 superclassed to N classes (Left), we find that the number of target classes does not have a visible effect on the scaling exponent. (Right) For CIFAR-10 with the addition of Gaussian noise to inputs, we find the strength of the noise has a strong effect on performance scaling with dataset size. All models are WRN-28-10.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 91 references

Research works printed but not linked: 4

  1. [9]J. Hoffmann, An empirical analysis of compute-optimal large language model training. Adv. Neural Inf. Process. Syst. 35, 30016– 30030 (2022).
  2. [14]A. Matthews, J. Hron, M. Rowland, R. E. Turner, Z. Ghahramani, “Gaussian process behaviour in wide deep neural networks” in International Conference on Learning Representations (2018).
  3. [30]M. S. Advani, A. M. Saxe, High-dimensional dynamics of generalization error in neural networks. arXiv [Preprint] (2017). 10.48550/arXiv.1710.03667 (Accessed 1 January 2021). PMC7685244 33022471 · DOI
  4. [44]D. de Laat, “Approximating manifolds by meshes: Asymptotic bounds in higher codimension” Thesis, University of Groningen, Groningen, The Netherlands (2011).

Cited by

Sorted by citations

See all 187

Related

Metadata

Authors
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U.
Affiliations
  • Google (United States) (US)
  • Johns Hopkins University (US)
Corresponding author
  • Bahri, Yasaman
  • Dyer, Ethan
  • Lee, Jaehoon
  • Sharma, Utkarsh
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
121
Issue
27
Pages
e2311878121-e2311878121
ISSN
0027-8424 (ISSN-L)
DOI
10.48550/arxiv.2102.06701
arXiv
2102.06701
PMC
PMC11228526
OpenAlex
W3133450183
Cited by
182
References
91
Citations by year
2026: 56 · 2025: 74 · 2024: 17 · 2023: 15 · 2022: 7 · 2021: 13
FWCI
21.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural scaling laws, power-law scaling, population decline, infinite-width limit, data manifold, random features, pre-trained models, scaling exponents, duality, model size, training data size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review