CoChat
Sign Up
PreprintCited by 427Machine Learning and Data ClassificationStochastic Gradient Optimization TechniquesAdvanced Neural Network Applications

Deep Learning Scaling is Predictable, Empirically

Open access
Hestness, Joel, Narang, Sharan, Ardalani, Newsha, Diamos, Gregory, Jun, Heewoo, Kianinejad, Hassan, Patwary, Md. Mostofa Ali, Yang, Yang +1 more·2017·arXiv
DOI 10.48550/arxiv.1712.00409arXiv 1712.00409
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1712.00409

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generalization error decreases as a power law in training-set size across several orders of magnitude, with task-dependent exponents and an irreducible error floor.

    “Hestness et al. [2017] conducted a large empirical study across machine translation, language modeling, image, and speech tasks, showing that generalization error decreases as a power law in training-set size across several orders of magnitude, with task-dependent exponents and an irreducible error floor.”

    From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Hestness et al., 2017

    The Introduction (p. 2) reports power-law improvement across machine translation, language modeling, image classification, and speech recognition, with different exponents by application. But §5.1 (p. 11) says the study had not reached the irreducible-error region in real applications; it describes that floor as likely and reports testing it only in toy problems. The claim overstates the floor as an empirical finding across the studied tasks. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • General methods that can leverage additional computation ultimately win out against methods that rely on human expertise.

    “The 'bitter lesson' of machine learning research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise (Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b; Keskar et al., 2019a).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Hestness et al., 2017

    The Introduction (p. 1) describes scaling data and computation as a lower-risk complement to creative model-architecture search, and the Conclusion (p. 13) says the resulting accuracy improvements are empirically predictable. The paper does not claim that computation-leveraging methods ultimately win over methods relying on human expertise. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • This is often guided by so-called scaling laws that predict the optimal model size and number of training steps based on data availability, incentivizing the development and deployment of increasingly large models as more data becomes available.

    “This is often guided by so-called scaling laws that predict the optimal model size and number of training steps based on data availability, incentivizing the development and deployment of increasingly large models as more data becomes available [49].”

    From From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate (2025) · cites this paper as [49]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Different methods can scale with different levels of efficiency, and successful methods must still satisfy some general requirements, such as being a consistent estimator and modeling pairwise statistics of P(Z).

    “Not quite: different methods can scale with different levels of efficiency (Hestness et al., 2017;Kaplan et al., 2020), and successful methods must still satisfy some general requirements (e.g., be a consistent estimator, model pairwise statistics of P(Z)).”

    From The Platonic Representation Hypothesis (2024) · cites this paper as Hestness et al., 2017

  • The paper's findings are very different from the unbridled exponential growth usually reported when just looking at the number of parameters of new deep learning models.

    “Our findings are very different from the unbridled exponential growth that is usually reported when just looking at the number of parameters of new deep learning models [Hestness et al., 2017, Kaplan et al., 2020, Henighan et al., 2020].”

    From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Hestness et al., 2017

    The Conclusion (p. 13) says model size scales sublinearly with training-data size, and Section 4.2.1 (p. 7) reports that pattern for tested language models. Those passages do not address the cited proposition: that parameter counts of newly introduced deep-learning models are usually reported to grow exponentially over time. Searches for exponential growth in model parameter counts found no passage making that point. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 33 · Cited by 427

Research works printed but not linked: 34

  1. S. Amari and N. Murata. Statistical Theory of Learning Curves under Entropic Loss Criterion. Neural Computation , 5(1):140-153, 1993.
  2. S. Amari, N. Fujita, and S. Shinomoto. Four Types of Learning Curves. Neural Computation , 4(4):605-618, 1992.
  3. D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. In Proceedings of The International Conference on Machine Learning (ICML) , pages 173-182, 2016.
  4. D. Arpit, S. Jastrz˛ ebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, and S. Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In Proceedings of the International Conference on Machine Learning , August 2017.
  5. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end Attention-based Large Vocabulary Speech Recognition. arXiv preprint arXiv:1508.04395v2 , 2016. · arXiv
  6. M. Banko and E. Brill. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proceedings of Association of Computational Linguistics (ACL) , January 2001.
  7. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. In Journal of Machine Learning Research 3 , pages 463-482, November 2002.
  8. E. Battenberg, J. Chen, R. Child, A. Coates, Y. Gaur, Y. Li, H. Liu, S. Satheesh, D. Seetapun, A. Sriram, and Z. Zhu. Exploring Neural Transducers for End-to-end Speech Recognition. arXiv preprint arXiv:1707.07413 , 2017. · arXiv
  9. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the Vapnik-Chervonenkis Dimension. Journal of the ACM (JACM) , 36(4):929-965, October 1989.
  10. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv preprint arXiv:1312.3005 , 2013. · arXiv
  11. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based Models for Speech Recognition. In Advances in Neural Information Processing Systems (NIPS) , pages 577-585, 2015.
  12. J. Collins, J. Sohl-Dickstein, and D. Sussillo. Capacity and Trainability in Recurrent Neural Networks. In Proceedings of The International Conference on Learning Representations (ICLR) , 2017.
  13. G. K. Dziugaite and D. M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Proceedings of The International Conference on Machine Learning (ICML) , 2017.
  14. A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A General Lower Bound on the Number of Examples Needed for Learning. Information and Computation , 82:247-261, 1989. Workshop on Computing Learning Theory, 1988.
  15. G. Györgyi and N. Tishby. Statistical Theory of Learning a Rule. Neural Networks and Spin Glasses , pages 3-36, 1990.
  16. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, et al. Deep Speech: Scaling Up End-to-End Speech Recognition. arXiv preprint arXiv:1412.5567 , 2014. · arXiv
  17. N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension Bounds for Piecewise Linear Neural Networks. In Proceedings of Machine Learning Research , volume 65, pages 1--12, 2017.
  18. D. Haussler. Quantifying Inductive Bias: AI Learning Algorithms and Valiant's Learning Framework. Artificial Intelligence , 36(2):177-221, 1988.
  19. D. Haussler, M. Kearns, H. S. Seung, and N. Tishby. Rigorous Learning Curve Bounds from Statistical Mechanics. Machine Learning , 25(2):195-236, November 1996.
  20. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410v2 , 2016. · arXiv
  21. K. Kawaguchi, L. P. Kaelbling, and Y. Bengio. Generalization in Deep Learning. arXiv preprint arXiv:1710.05468v1 , October 2017. · arXiv
  22. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of the Association for Computational Linguistics (ACL) , 2017.
  23. T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1412-1421, 2015.
  24. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575 , January 2015. · arXiv
  25. H. Schwarze and J. Hertz. Statistical Mechanics of Learning in a Large Committee Machine. In Advances in Neural Information Processing Systems, (NIPS) , pages 523-530, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc.
  26. R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909 , 2016a. · arXiv
  27. R. Sennrich, B. Haddow, and A. Birch. Edinburgh Neural Machine Translation Systems for WMT 16. arXiv preprint arXiv:1606.02891 , 2016b. · arXiv
  28. H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical Mechanics of Learning from Examples. Physical Review A , 45:6056-6091, April 1992.
  29. S. L. Smith and Q. V. Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. arXiv preprint arXiv:1710.06451v2 , October 2017. · arXiv
  30. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.
  31. H. G. Tucker. A Generalization of the Glivenko-Cantelli Theorem. The Annals of Mathematical Statistics , 30 (3):828-830, September 1959.
  32. V. Vapnik. An Overview of Statistical Learning Theory. In IEEE Transactions on Neural Networks , volume 10, pages 988-999, September 1998.
  33. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding Deep Learning Requires Rethinking Generalization. arXiv preprint arXiv:1611.03530v2 , 2017. · arXiv
  34. J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent Highway Networks. In Proceedings of The International Conference on Machine Learning (ICML) , 2017.

Authors
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1712.00409
arXiv
1712.00409
OpenAlex
W2775461895
Cited by
427
References
33
Citations by year
2026: 12 · 2025: 60 · 2024: 60 · 2023: 74 · 2022: 52 · 2021: 70 · 2020: 50 · 2019: 37 · 2018: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
generalization error, training data size, model size, power-law scaling, learning curve, sublinear scaling, model accuracy, machine translation, language modeling, image processing, speech recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review