The DOI matches its Crossref record; 155 of the 155 references it cites were found in a registry or an index; 4 of the 43 sources its claims cite were checked for support.
Compute Trends Across Three Eras of Machine Learning
Open accessAbstract
Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
Overview
CoChat Summary
TL;DR
The paper curates a dataset of 123 milestone ML systems and finds that training compute grew slowly before 2010, accelerated around the advent of Deep Learning, and then split into a separate large-scale trend from late 2015. It frames these changes as three eras: Pre Deep Learning, Deep Learning, and Large-Scale.
Key findings
- Pre Deep Learning Era (1952–2010): training compute doubled roughly every 21.3 months (95% CI [17.0; 21.2; 29.3]) for all models, growing in line with Moore’s law; the conclusion summarizes this as an 18-month doubling time.
- Deep Learning Era (2010–2022): the trend accelerated to about a 5.7-month doubling time for regular-scale models (Table 2), with log-linear regressions giving 5.6 months for all models 2010–2022 (95% CI [5.0; 5.6; 6.2]) and 5.7 months for 2012–2022.
- Large-Scale Era (late 2015–2022): a new trend of large-scale models emerged, starting 2 to 3 orders of magnitude above the previous trend and doubling every 9.9 months (Table 2) or 10.7 months for 2016–2022 (Table 4); the regular-scale trend continued at 5–6 months.
- The paper attributes discrepancies with prior work—such as Amodei & Hernandez’s 3.4-month doubling (2012–2018) and Lyzhov’s >2-year doubling (2018–2020)—to limited samples and single-trend assumptions, whereas this analysis separates large-scale and regular-scale models.
Methods
- Curated 123 milestone ML systems from papers meeting necessary criteria (explicit learning component, experimental results, state-of-the-art advance) and at least one notability criterion (>1000 citations, historical importance, important SotA advance, or deployed in a notable context), drawing on literature reviews, Papers With Code, historical accounts, previous datasets, most-cited publications, and suggestions.
- Estimated training compute when not reported using techniques from AI and Compute’s appendix, including forward-pass compute and GPU time; estimates are annotated, and only the final training run is counted, not hyperparameter tuning.
- Used log-linear regressions on training compute to derive doubling rates; confidence intervals come from a bootstrap with B = 1000 samples and random adjustment of each estimate by a factor between 1/2 and 2; low-compute outliers were excluded via Z-score < −2 relative to models within 1.5 years, and large-scale models were selected with a Z threshold (e.g., Z ≥ 0.76 after 2016).
Limitations
- The paper states the dataset is biased in important ways and likely contains mistakes; compute estimates are expected to be accurate within a factor of about two, with non-sampling errors possible, and only final training runs are annotated.
- The paper states selection criteria are subjective and biased toward academic, English-language, and subjectively “notable” models—more likely large and recent—and that small sample size could affect results; it also says the distinction between large-scale and regular-scale models is not confident, was made intuitively and justified post hoc, and that limited data on large-scale models means the apparent slowdown might be noise.
- The text makes evident that results vary across analyses and thresholds (e.g., n = 121 in Figure 1, n = 102 in Figure 3, different Z thresholds in Table 9), and the paper itself notes there is room for alternative interpretations of the data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 5 Not valid · 27 Not checked · 71 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Various researchers have highlighted the relationship between AI capabilities and the scaling of ML models (Kaplan et al., 2020; Sutton, 2019; Z. Li et al., 2020; Jones, 2021; Rosenfeld et al., 2019; Hestness et al., 2017).”
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
The Summary (p. 2) reports that language-model performance improves with model size, dataset size, and training compute. But the Discussion (p. 19) says it is not yet known whether these scaling results generalize beyond language models or whether lower loss translates into better performance on relevant language tasks. The claim’s reference to AI capabilities broadly exceeds what the paper establishes. (LLM, conf=0.85)
- Not matchedcites Sutton, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Z. Li et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, compute can be seen as a quantifiable proxy for the progress of ML research.”
What the sources say
- Partly supportedcites Kaplan et al., 2020Experimental — accuracy not yet measured
The Summary (pp. 2–3) reports a predictable relationship between training compute and Transformer language-model performance. It does not establish compute as a proxy for progress across ML research, and it says model size and data must also be scaled for optimal performance. (LLM, conf=0.84)
- Not matchedcites Sutton, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Z. Li et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, Sharir et al. (2020) estimates that Google's T5 project-which is by no means the biggest training run to date-might have cost a total of $10M in cloud computing costs.”
What the sources say
- Supportedcites Sharir et al. (2020) source
In “Costs: Not for the faint hearted” (p. 1), Sharir et al. estimate that the list-price cost of the entire T5 project may have been $10 million. The accompanying note says the figures assume cloud services such as GCP or AWS, matching the claim’s description of cloud computing costs. (LLM, conf=0.93)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The hardware constraints involved in massive training runs (including memory limitations and communication bandwidths) force users to segment massive models into groups of layers, which are then trained in parallel (Hazelwood et al., 2018; Huang et al., 2019; Athlur et al., 2021).”
What the sources say
- Not checked — no full text of the sourcecites Hazelwood et al., 2018 source
- Not matchedcites Huang et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Athlur et al., 2021 sourceExperimental — accuracy not yet measured
The “Model-parallel Training” and “Inter-layer or pipeline parallelism” sections say that distributing layers across GPUs helps fit massive models and enables pipeline training. The paper also discusses GPU memory limits and communication bandwidth, but it does not say these constraints force users to partition models into layer groups; it describes other approaches and automatic partitioning. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These large-scale models were trained by large corporations, whose larger training budgets presumably enabled them to break the previous trend.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
41 more claims
Create a free account to see all 107 claims and every source verdict.
- It may state two separate claims as one. 33
- Probably not a substantial claim. 37
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Between the release of GPT-2 and today, the compute used to train the most impressive frontier language models has increased by at least a factor of 10,000x [40], with training compute usage doubling every 6-10 months [41].”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [41]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Methods · p. 16
“This criteria results in the exclusion of 5 models out of 123 between 1952 and 2022.”
§ Training compute (FLOPs) of milestone Machine Learning systems over time · p. 17
“Figure 4: The low compute outliers ( n = 7 ) are highlighted in green.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Methods · p. 16
“we compute the log training compute Z -score of each model with respect to other models whose publication date is within 1.5 years.”
§ A Methods · p. 16
“The Z value represents the distance of the log compute of each system relative to the mean of systems published within 2 years of the paper in question, normalized by the standard deviation.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Training compute (FLOPs) of milestone Machine Learning systems over time · p. 3
“| 2010 to 2022 | Regular-scale models | 7e+14 to 2e+18 FLOPs | 0.6 OOMs/year | 5.7 months |”
§ Training compute (FLOPs) of milestone Machine Learning systems over time · p. 4
“| 2010-2022 | Regular-scale ( n = 77 ) | 4e+14 / 2e+22 | 0.7 OOMs/year [0.6; 0.7; 0.7] | 5.6 months [5.1; 5.6; 6.2] | 0.78 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ E Comparison to Amodei & Hernandez's analysis · p. 20
“The period starts with AlexNet in September 2012 and ends with AlphaZero in December 2018.”
§ E Comparison to Amodei & Hernandez's analysis · p. 21
“| AlexNet to AlphaZero 09-2012 to 12-2017 | All models ( n = 31 )”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 Conclusion · p. 6
“we identify an 18-month doubling time between 1952 and 2010”
§ Training compute (FLOPs) of milestone Machine Learning systems over time · p. 3
“| 1952 to 2010 | All models ( n = 19 ) | 3e+04 to 2e+14 FLOPs | 0.2 OOMs/year | 21.3 months |”
2 more findings
Create a free account to see every finding of this check.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Research works printed but not linked: 50
- ba, 2016, Layer Normal-ization
- leahy, 2022, Announcing GPT-NeoX-20B
- sevilla, 2021, emphParame-ter Counts in Machine Learning
- li, 2020, OpenAI's GPT-3 language model A technical overview
- deng, 2009, Deep Learning for Speech Recognition and Related Applications
- wang, 2020, DeepMind achieved StarCraft II GrandMas-ter Level but at what cost?
- h, 2020, How Much Did AlphaGo Zero Cost
- desislavov, 2021, Compute and Energy Consumption Trends in Deep Learning Inference
- moore, 1965, emphThe Future of Integrated Electronics
- sevilla, 2022, Estimating training compute of Deep Learning models
- 0, Hardware- Und Nachrichten-Links Des 30./31. Okto-ber 2021, 3D Center
- komatsuzaki, 2021, GPT-J-6B 6B JAX-Based Trans-former
- 0, Naver Corporation, NVE Corporation
- nad, 0, DALL E Creating Images from Text
- 0, GPT-Neo, Eleuther AI
- barrett, 2021, Taiwan's drought is exposing just how much water chipmakers like TSMC use (and reuse)
- orme, 2022, Report Microsoft Handed OpenAI $500m in Azure Credits
- 10.1109/CVPR.2001.990517 · DOI · Paper page
- 2021, OpenAI API Pricing, OpenAir
- krizhevsky, 0, ImageN et Classification with Deep Convolutional Neural Networks, Advances in neural information processing systems, 25
- kohs, 2017, AlphaGo Moxie Pictures Reel As Dirt
- rae, 0, Language modelling at scale
- alvi, 2021, Using DeepSpeed and Megatron to Train Megatron- Turing NLG 530B the World's Largest and Most Powerful Generative language Model
- lieber, 0, Announcing AI21 Studio and Jurassic-1 Language Models
- amodei, 2018, AI and Compute
- lyzhov, 2021, AI and Compute Trend Isn't Predictive of What Is Happening
- sastry, 2019, AI and Compute Addendum Compute Used in Older Headline Results
- 0, Baidu Research, Baidu Res
- 10.1038/s41586-019-1724-z · DOI · Paper page
- sutton, 2019, The bitter lesson, Incomplete Ideas (blog), 13, 12
- wiggers, 2021, Google Trained a Trillion-Parameter AI Language Model
- ajmera, 2021, Ford to Shut Some N American Plants for Few Weeks on Chip Shortage
- shilov, 2020, GPU Shortages Hit Nvidia's Data Center Business Not Enough $15 000+ GPUs
- 2019, AlphaStar Grandmaster level in StarCraft II using multiagent reinforcement learning
- cire?an, 2010, Deep, Big, Simple Neural Nets for Handwritten Digit Recognition, Neural Computation, 22
- radford, 2019, Better Language Models and Their Implications
- 0, Solving rubik's cube with a robot hand
- adiwardana, 2020, Towards a Conversational Agent that Can Chat About Anything
- rosset, 2020, Turing-NLG A 17-Billion-Parameter Language Model by Microsoft
- woodie, 2021, The Chip Shortage Seems to Be Impacting AI Workloads in the Cloud
- patel, 2021, Why the Global Chip Shortage is Making It So Hard to Buy a PS5
- chen, 2020, Image GPT
- klein, 0, Mighty mouse
- selfridge, 0, Pandemonium A Paradigm for Learning |AITopics
- dosovitskiy, 2020, An image is worth 16×16 words Transformers for image recognition at scale
- lohn, 2022, How Much Longer Can Com-puting Power Drive Artificial Intelligence Progress?, Center for Security and Technology Tech Rep
- 2022, The Akronomicon - LightOn AI Research, Akro-nomicon
- 2022, Computer Progress, Comput Program
- 2022, AI Tracker, AI Tracker
- radford, 2018, Improving language understanding with unsupervised learning
Cited by
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
Harrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480
Experimentally realized in situ backpropagation for deep learning in photonic neural networks
Pai, S., Sun, Z., Hughes, T. W., Park, T., Bartlett, B., Williamson, I. A. D., Minkov, M., Milanizadeh, M., Abebe, N., Morichetti, F., et al. (2023) · Science · doi:10.1126/science.ade8450 · Cited by 331
Generative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293
Shoaib, M., Hussain, T., Shah, B., Ullah, I., Shah, S. M., Ali, F., & Park, S. H. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.1031748 · Cited by 272
“The writers used the MnasNet network (Sevilla et al., 2022) to create their baseline architecture, prioritizing model accuracy and FLOPA using a neural network architecture to search multi-objects.”
Method
Metadata
- Authors
- Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P.
- Affiliations
- University of Aberdeen (GB)
- Centre for the Governance of AI (GB)
- Epoch
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- 2022 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2022-07-18
- Pages
- 1-8
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W4312968147
- Cited by
- 363
- References
- 153
- Citations by year
- 2026: 57 · 2025: 92 · 2024: 111 · 2023: 87 · 2022: 16
- FWCI
- 61.54
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- deep learning, large-scale machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Compute Trends Across Three Eras of Machine Learning. https://app.cochat.ai/papers/10.1109/ijcnn55064.2022.9891914
- The paper
- Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022). Compute Trends Across Three Eras of Machine Learning. 2022 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/ijcnn55064.2022.9891914
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.