Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 155 of the 155 references it cites were found in a registry or an index; 4 of the 43 sources its claims cite were checked for support.

Compute Trends Across Three Eras of Machine Learning

Open access
Sevilla, Jaime, Heim, Lennart, Ho, Anson, Besiroglu, Tamay, Hobbhahn, Marius, Villalobos, Pablo · 2022 · 2022 International Joint Conference on Neural Networks (IJCNN)

Abstract

Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

Overview

CoChat Summary

TL;DR

The paper curates a dataset of 123 milestone ML systems and finds that training compute grew slowly before 2010, accelerated around the advent of Deep Learning, and then split into a separate large-scale trend from late 2015. It frames these changes as three eras: Pre Deep Learning, Deep Learning, and Large-Scale.

Key findings

  • Pre Deep Learning Era (1952–2010): training compute doubled roughly every 21.3 months (95% CI [17.0; 21.2; 29.3]) for all models, growing in line with Moore’s law; the conclusion summarizes this as an 18-month doubling time.
  • Deep Learning Era (2010–2022): the trend accelerated to about a 5.7-month doubling time for regular-scale models (Table 2), with log-linear regressions giving 5.6 months for all models 2010–2022 (95% CI [5.0; 5.6; 6.2]) and 5.7 months for 2012–2022.
  • Large-Scale Era (late 2015–2022): a new trend of large-scale models emerged, starting 2 to 3 orders of magnitude above the previous trend and doubling every 9.9 months (Table 2) or 10.7 months for 2016–2022 (Table 4); the regular-scale trend continued at 5–6 months.
  • The paper attributes discrepancies with prior work—such as Amodei & Hernandez’s 3.4-month doubling (2012–2018) and Lyzhov’s >2-year doubling (2018–2020)—to limited samples and single-trend assumptions, whereas this analysis separates large-scale and regular-scale models.

Methods

  • Curated 123 milestone ML systems from papers meeting necessary criteria (explicit learning component, experimental results, state-of-the-art advance) and at least one notability criterion (>1000 citations, historical importance, important SotA advance, or deployed in a notable context), drawing on literature reviews, Papers With Code, historical accounts, previous datasets, most-cited publications, and suggestions.
  • Estimated training compute when not reported using techniques from AI and Compute’s appendix, including forward-pass compute and GPU time; estimates are annotated, and only the final training run is counted, not hyperparameter tuning.
  • Used log-linear regressions on training compute to derive doubling rates; confidence intervals come from a bootstrap with B = 1000 samples and random adjustment of each estimate by a factor between 1/2 and 2; low-compute outliers were excluded via Z-score < −2 relative to models within 1.5 years, and large-scale models were selected with a Z threshold (e.g., Z ≥ 0.76 after 2016).

Limitations

  • The paper states the dataset is biased in important ways and likely contains mistakes; compute estimates are expected to be accurate within a factor of about two, with non-sampling errors possible, and only final training runs are annotated.
  • The paper states selection criteria are subjective and biased toward academic, English-language, and subjectively “notable” models—more likely large and recent—and that small sample size could affect results; it also says the distinction between large-scale and regular-scale models is not confident, was made intuitively and justified post hoc, and that limited data on large-scale models means the apparent slowdown might be noise.
  • The text makes evident that results vary across analyses and thresholds (e.g., n = 121 in Figure 1, n = 102 in Figure 3, different Z thresholds in Table 9), and the paper itself notes there is room for alternative interpretations of the data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/ijcnn55064.2022.9891914

Cites

155 of 155 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 5 Not valid · 27 Not checked · 71 Own results and interpretations

41 more claims

Create a free account to see all 107 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 33
  • Probably not a substantial claim. 37
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 7 · the rest aren’t loaded

2 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 153 references

Research works printed but not linked: 50

  1. ba, 2016, Layer Normal-ization
  2. leahy, 2022, Announcing GPT-NeoX-20B
  3. sevilla, 2021, emphParame-ter Counts in Machine Learning
  4. li, 2020, OpenAI's GPT-3 language model A technical overview
  5. deng, 2009, Deep Learning for Speech Recognition and Related Applications
  6. wang, 2020, DeepMind achieved StarCraft II GrandMas-ter Level but at what cost?
  7. h, 2020, How Much Did AlphaGo Zero Cost
  8. desislavov, 2021, Compute and Energy Consumption Trends in Deep Learning Inference
  9. moore, 1965, emphThe Future of Integrated Electronics
  10. sevilla, 2022, Estimating training compute of Deep Learning models
  11. 0, Hardware- Und Nachrichten-Links Des 30./31. Okto-ber 2021, 3D Center
  12. komatsuzaki, 2021, GPT-J-6B 6B JAX-Based Trans-former
  13. 0, Naver Corporation, NVE Corporation
  14. nad, 0, DALL E Creating Images from Text
  15. 0, GPT-Neo, Eleuther AI
  16. barrett, 2021, Taiwan's drought is exposing just how much water chipmakers like TSMC use (and reuse)
  17. orme, 2022, Report Microsoft Handed OpenAI $500m in Azure Credits
  18. 10.1109/CVPR.2001.990517 · DOI · Paper page
  19. 2021, OpenAI API Pricing, OpenAir
  20. krizhevsky, 0, ImageN et Classification with Deep Convolutional Neural Networks, Advances in neural information processing systems, 25
  21. kohs, 2017, AlphaGo Moxie Pictures Reel As Dirt
  22. rae, 0, Language modelling at scale
  23. alvi, 2021, Using DeepSpeed and Megatron to Train Megatron- Turing NLG 530B the World's Largest and Most Powerful Generative language Model
  24. lieber, 0, Announcing AI21 Studio and Jurassic-1 Language Models
  25. amodei, 2018, AI and Compute
  26. lyzhov, 2021, AI and Compute Trend Isn't Predictive of What Is Happening
  27. sastry, 2019, AI and Compute Addendum Compute Used in Older Headline Results
  28. 0, Baidu Research, Baidu Res
  29. 10.1038/s41586-019-1724-z · DOI · Paper page
  30. sutton, 2019, The bitter lesson, Incomplete Ideas (blog), 13, 12
  31. wiggers, 2021, Google Trained a Trillion-Parameter AI Language Model
  32. ajmera, 2021, Ford to Shut Some N American Plants for Few Weeks on Chip Shortage
  33. shilov, 2020, GPU Shortages Hit Nvidia's Data Center Business Not Enough $15 000+ GPUs
  34. 2019, AlphaStar Grandmaster level in StarCraft II using multiagent reinforcement learning
  35. cire?an, 2010, Deep, Big, Simple Neural Nets for Handwritten Digit Recognition, Neural Computation, 22
  36. radford, 2019, Better Language Models and Their Implications
  37. 0, Solving rubik's cube with a robot hand
  38. adiwardana, 2020, Towards a Conversational Agent that Can Chat About Anything
  39. rosset, 2020, Turing-NLG A 17-Billion-Parameter Language Model by Microsoft
  40. woodie, 2021, The Chip Shortage Seems to Be Impacting AI Workloads in the Cloud
  41. patel, 2021, Why the Global Chip Shortage is Making It So Hard to Buy a PS5
  42. chen, 2020, Image GPT
  43. klein, 0, Mighty mouse
  44. selfridge, 0, Pandemonium A Paradigm for Learning |AITopics
  45. dosovitskiy, 2020, An image is worth 16×16 words Transformers for image recognition at scale
  46. lohn, 2022, How Much Longer Can Com-puting Power Drive Artificial Intelligence Progress?, Center for Security and Technology Tech Rep
  47. 2022, The Akronomicon - LightOn AI Research, Akro-nomicon
  48. 2022, Computer Progress, Comput Program
  49. 2022, AI Tracker, AI Tracker
  50. radford, 2018, Improving language understanding with unsupervised learning

Cited by

See all 363

Metadata

Authors
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P.
Affiliations
  • University of Aberdeen (GB)
  • Centre for the Governance of AI (GB)
  • Epoch
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
Venue
2022 International Joint Conference on Neural Networks (IJCNN)
Publisher
IEEE
Published
2022-07-18
Pages
1-8
ISSN
2161-4407 (ISSN-L)
OpenAlex
W4312968147
Cited by
363
References
153
Citations by year
2026: 57 · 2025: 92 · 2024: 111 · 2023: 87 · 2022: 16
FWCI
61.54
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
deep learning, large-scale machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Compute Trends Across Three Eras of Machine Learning. https://app.cochat.ai/papers/10.1109/ijcnn55064.2022.9891914
The paper
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022). Compute Trends Across Three Eras of Machine Learning. 2022 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn55064.2022.9891914

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.