The DOI matches its DataCite record.
Training Compute-Optimal Large Language Models
Open accessAbstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, whereas recent advances in modeling and hardware have increased the data training capacity of LLMs, increasing from Wikipedia-scale corpora to close to three orders of magnitude more”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as Hoffmann et al. [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The costs required to create such models (and continuing returns to scale) means that only well-resourced actors can create cuttingedge models (Epoch, 2023; Hoffmann et al., 2022; Kaplan et al., 2020), making them relatively few in number.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite containing 405B parameters, our largest Llama 3 in fact contains fewer parameters than earlier and much less performant models such as PALM (Chowdhery et al., 2023), due to better understanding of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).”
From The Llama 3 Herd of Models (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given that deep learning AI and LLMs AI models exhibit similar power scaling laws, our focus can be directed specifically towards LLMs AI without a significant loss of generality.”
From Large language models at work in China’s labor market (2025) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work (Kaplan et al., 2020;Hoffmann et al., 2022) assumes that the necessary data to support scaling is unlimited.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is very close to the 73 billion parameters and 1.3 trillion tokens predicted by the IsoFLOP curves on C4 from (Hoffmann et al., 2022)”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on their IsoFLOP training runs on C4, they only provide the information that for C4, compute (C) allocated to data (D) and parameters (N ) should be scaled exactly equally for optimality, i.e. a = b = 0.5”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the precise ratios vary considerably across data sets and approaches.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example given the Gopher (Rae et al., 2021) compute budget of 5.76 × 10 23 FLOPs, their parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, if the training data set is C4 (Raffel et al., 2020) their IsoFLOP approach predicts 73 billion parameters to be optimal, almost twice as much.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The creators of Galactica decided to train a 120 billion parameter model on 450 billion tokens, a significant overallocation to parameters even in Chinchilla terms (black efficient frontier).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, the default Chinchilla scaling law (Hoffmann et al., 2022) predicts loss to continue decreasing as parameters are added, which is in stark contrast to the empirical data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In many cases, the effect of scale on performance can often be methodologically predicted via scaling laws-for example, scaling curves for cross-entropy loss have been shown to empirically span more than seven orders of magnitude (Kaplan et al., 2020; Hoffmann et al., 2022).”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For GPT-3, Gopher, and Chinchilla, models of ∼ 10 22 training FLOPs ( ∼ 10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to 3-5 · 10 23 training FLOPs (70B-280B parameters) enables performance to substantially surpass random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Hoffmann et al.'s estimated model L(N, D) = 1.6934 + 406.4 N 0.3392 + 410.7 D 0.2849”
From Chinchilla Scaling: A replication attempt (2024) · cites this paper as Hoffmann et al. (2022)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture (Hoffmann et al. 2022; Rae et al. 2021).”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [50]
The cited source says what the claim attributes to it.
In the citing paper
“While AI systems focus on scaling neural, data, and computation capacity (30, 31, 74–77), our findings suggest that evolution of embodied intelligent behavior (biological or artificial) may require coordinated scaling of visual acuity and processing.”
From What if eye...? Computationally recreating vision evolution (2025) · cites this paper as [74]
The cited source says what the claim attributes to it.
In the citing paper
“Recent investigations on large language models have profiled the trade-off between model size and data volume when the compute budget is fixed42, suggesting model size is another important dimension for pre-training worthy of further investigation for human sensing data.”
From Self-supervised learning for human activity recognition using 700,000 person-days of wearable data (2024) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“The capacity and capability of LLMs greatly increased with scale, taking the same path that nature took by evolving bigger and better brains (Allman, 1999; Hoffmann et al., 2022).”
From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“recent work has put a strong emphasis on the quantity of data being a strong factor in a trained model's performance on evaluation tasks (Kaplan et al., 2020; Hoffmann et al., 2022)”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Scaling laws revealed predictable relationships between parameter count, data volume, and loss, reinforcing the intuition that progress tracked directly with model size [Kaplan et al., 2020, Hoffmann et al., 2022].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“The impact of scaling is illustrated by the scaling laws [34, 35], which numerically uncover the power-law relationship between model size, data volume, training scale and the growth of the model's performance.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“With increased model size, given adequate data and computational resources, proficiency in word prediction within textual contexts, as demonstrated by the training dataset, is enhanced (Kaplan et al. 2020; Ganguli et al. 2022; Hoffmann et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“Hoffmann et al. [24] models the improvement for larger model training budgets as a power function, proportional to the model parameters N and training tokens D , with the exact functional form and prediction setup varying between work [44].”
From Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (2025) · cites this paper as [24]
The cited source says what the claim attributes to it.
In the citing paper
“Notably, the 70-billion parameter Chinchilla model (Hoffmann et al., 2022) outperforms the 280-billion parameter Gopher model (Rae et al., 2021) while using a similar compute budget by being trained on four times more data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Allocation is balanced: resources are divided roughly equally between scaling of parameters and data (Hoffmann et al., 2022).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“These methods lead to the conclusion that α ≈ β and hence N and D should be scaled proportionally for compute-optimal training.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While Hoffmann et al. (2022) have shown that the equal scaling of model parameters and training tokens holds across different training data sets”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“However, for C4, which is our training data set, they do not provide the coefficients necessary to compute loss with their parametric loss function.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Results from (Hoffmann et al., 2022) show that there are differences in the scaling coefficients when training on C4 as compared to GitHub code, however, the overall allocation trend is the same.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“We find the parameter counts reported in Chinchilla (Hoffmann et al., 2022) to be significantly different than our calculations, especially at larger scales.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“they report a confidence intervals of 0.454 to 0.455 and 0.542 to 0.543 for a and b respectively.”
From Chinchilla Scaling: A replication attempt (2024) · cites this paper as Hoffmann et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law.”
From Chinchilla Scaling: A replication attempt (2024) · cites this paper as Hoffmann et al. (2022)
The cited source says what the claim attributes to it.
In the citing paper
“The starting point is the scaling law from Hoffmann et al. 2022, which relates the training loss L of a dense transformer to its number of parameters N and the training dataset size D :”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“given that that the physical compute is approximately given by C ≈ 6 ND (Hoffmann et al. 2022; Kaplan et al. 2020)”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“By directly applying the new scaling law, we calculate it provides a 2 × to 4 × reduction in compute needed to reach a given loss target at the scale of current frontier LLMs, depending on the scale of the model (see Appendix B).”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“The rapid scaling of compute for training language models (Sevilla et al. 2022), coupled with insights from scaling laws (Hoffmann et al. 2022; Kaplan et al. 2020), suggests that a substantial portion of the improvement in language model capabilities can be attributed to the increased use of computational resources.”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“This is in spite of the estimate by Hoffmann et al. 2022 that N/D should be O (1) for compute-optimal training.”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“we have the approximate relation that C ≈ 6 ND (Hoffmann et al. 2022).”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Notably, GPT-3 and Chinchilla fail to achieve one-shot performance of better than random, even when scaled to their largest model size of ∼ 5 · 10 23 FLOPs.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In comparison, Kaplan et al. 2020 find a scaling exponent of around -0.048, and Hoffmann et al. 2022 estimate values of around -0.3.”
From Algorithmic progress in language models (2024) · cites this paper as Hoffmann et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
This paper’s reference list could not be split into single entries.
Cited by
Sorted by citations
Items 101–116 of 116
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
Chinchilla Scaling: A replication attempt
Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024) · arXiv · doi:10.48550/arxiv.2404.10102 · Cited by 1
Li, H., Zheng, W., Wang, Q., Zhang, H., Wang, Z., Xuyang, S., Fan, Y., Ding, Z., Wang, H., Ding, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.04715 · Cited by 1
Alviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
Towards Embodiment Scaling Laws in Robot Locomotion
Ai, B., Dai, L., Bohlinger, N., Li, D., Mu, T., Wu, Z., Fay, K., Christensen, H. I., Peters, J., & Su, H. (2025) · arXiv · doi:10.48550/arxiv.2505.05753
Gao, X., Xie, W., Lixin, Fang, X., Hang, C., Li, C., Lin, Y., & Xu, X. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i25.39266
The AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
Scaling Data-Constrained Language Models with Synthetic Data
Kiyomaru, H., Oda, Y., Kodama, T., Liu, C., & Kawahara, D. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.52
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data
Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2022) · arXiv · doi:10.48550/arxiv.2206.02909
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citation
Metadata
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., & Sifre, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2203.15556
- OpenAlex
- W7133225605
- Cited by
- 929
- References
- 0
- Citations by year
- 2026: 239 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26
- FWCI
- 96.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
