Training Compute-Optimal Large Language Models
Open accessThe DOI matches its Crossref record; 53 of the 78 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Overview
CoChat Summary
TL;DR
The paper estimates compute-optimal scaling for transformer language models and finds that model size and training tokens should be increased in roughly equal proportions as compute grows. It validates this by training Chinchilla, a 70B model on 1.4T tokens, which outperforms much larger models such as Gopher (280B) under the same compute budget.
Key findings
- Across three fitting approaches, the scaling exponents for optimal parameters \(N_{opt} \propto C^a\) and tokens \(D_{opt} \propto C^b\) are near equal: Approach 1 gives \(a=0.50, b=0.50\); Approach 2 gives \(a=0.49, b=0.51\); Approach 3 gives \(a=0.46, b=0.54\), in contrast to Kaplan et al. (2020), which reported \(a=0.73, b=0.27\).
- Chinchilla (70B parameters, 1.4T tokens) uses the same compute budget as Gopher (280B, 300B tokens) and outperforms Gopher, GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream tasks.
- Chinchilla reaches 67.6% average 5-shot accuracy on MMLU, improving on Gopher by 7.6%; it also improves BIG-bench average accuracy to 65.1% versus 54.4% for Gopher, and achieves 77.4% on LAMBADA zero-shot versus 74.5% for Gopher.
- The paper concludes that current large language models are significantly undertrained and that a 175B model should be trained on over 4.2 trillion tokens, while a 280B Gopher-like model should be trained on 6.8 trillion tokens.
Methods
- The authors train over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, then fit scaling relationships using three approaches: fixing model sizes and varying training tokens, IsoFLOP profiles, and fitting a parametric loss function.
- They use the resulting compute-optimal frontier to train Chinchilla, a 70B parameter autoregressive transformer, on 1.4 trillion tokens using the same compute budget as Gopher, with architecture and training setup based on Gopher but with changes including AdamW, a modified SentencePiece tokenizer without NFKC normalization, and a different MassiveText subset distribution.
- Evaluation covers language modelling, reading comprehension, question answering, common sense, MMLU, BIG-bench, gender bias, and toxicity, largely following the evaluation setup of Rae et al. (2021).
Limitations
- The paper states that due to the cost of training large models, it has only two comparable large-scale training runs (Chinchilla and Gopher) and no additional tests at intermediate scales.
- It assumes the efficient computational frontier follows a power-law relationship, but observes concavity in \(\log N_{opt}\) at high compute budgets, suggesting it may still overestimate the optimal size of large models.
- All training runs for the analysis used less than one epoch of data, so future work may consider the multiple-epoch regime; the paper also notes that language modelling comparisons may be affected by train/test leakage because Chinchilla is trained on 4× more data than Gopher, and its bias/toxicity evaluations are not comprehensive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
22 Valid · 5 Not valid · 9 Not checked · 88 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Section D.4, we show a head-to-head comparison at 10 21 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)-using the model size we predict has a clear advantage.”
What the sources say
- Supportedcites Kaplan et al. (2020)
Section 6.1 (p. 15) gives Kaplan et al.’s recommended model size as a function of compute budget, so their analysis can supply a model size for a comparison at 10^21 FLOPs. The claimed advantage of the citing authors’ model size is presented as the citing authors’ own finding, not as a finding attributed to Kaplan et al. (LLM, conf=0.90)
boostedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 2 (Robbins and Monro, 1951) (and may attain the bound).”
What the sources say
- Could not verifycites Robbins and Monro, 1951 sourceExperimental — accuracy not yet measured
The Summary (p. 1) and Convergence theorems (pp. 2–5) establish convergence of a stochastic approximation procedure, but the readable passages do not give a 1/√D lower bound for the convergence rate of stochastic first-order methods. I searched for the rate, lower bound, and efficiency; several formulas in our copy are undecoded, so the claimed bound cannot be verified from it. (LLM, conf=0.72)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.”
What the sources say
- Supportedcites Izacard and Grave, 2020 source
Table 2 (4.2 Technical Details, p. 7) reports a 72.5 TriviaQA test exact-match score for the paper’s open-book retriever–reader system, and 4.3 Results (p. 7) describes it as outperforming the prior state of the art. The citation supports the open-book SOTA benchmark it identifies; the 7.9-point comparison with Chinchilla is the citing authors’ calculation, not a result reported in this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
What the sources say
- Supportedcites Kaplan et al. (2020)
The citation identifies Kaplan et al.’s approach and results, rather than asking that paper to assess the citing authors’ approach. Section 3 (p. 7) describes the models Kaplan et al. trained, and the Discussion (p. 18) states their scaling results. The comparison with the citing authors’ results is their own conclusion. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.”
What the sources say
- Partly supportedcites Kaplan et al. (2020)Experimental — accuracy not yet measured
Section 6.1 (p. 15) says that each compute budget has an associated optimal model size and that the authors fit this relationship with a power law. Our copy does not give a readable coefficient or a value at 10^21 FLOPs, so it does not verify the specific figure of 4.68 billion parameters. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
45 more claims · 2 with their cited sources checked
Create a free account to see all 124 claims and to verify the 2 references still open.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 34
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Initially, scaling efforts focused on training-time expansion through increases in model and dataset size [14,18].”
From A Survey of Large Language Models (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Touvron et al. (2023a) are forced to pull numbers from the PaLM and Chinchilla technical reports (Chowdhery et al., 2023; Hoffmann et al., 2022) directly for baseline comparison, with no option to rerun these models on Llama's evaluation implementations since PaLM and Chinchilla are not publicly available.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformers are widely known to under-perform with insufficient training compute compared to RNNs [53, 40].”
From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measured
22 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Training Compute-Optimal Large Language Models · p. 1
“Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark”
§ 4.2.1. Language modelling · p. 11
“| Chinchilla 5-shot | 67.6% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2.5. Common sense · p. 12
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks”
§ 4.2.7. Gender bias and toxicity · p. 13
“| PIQA | 81.8% | 81.8% | 81.0% | 82.0% | 90.1% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 175 Billion | 4.54e+24 | 4.3 Trillion | 1.26e+24 | 12.0 Trillion |”
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 67 Billion | 6.88e+23 | 1.7 Trillion | 1.71e+24 | 4.1 Trillion |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4. Optimal model scaling · p. 8
“Unless one has a compute budget of 10 26 FLOPs (over 250 the compute used to train Gopher )”
§ 3.4. Optimal model scaling · p. 8
“| 1 Trillion | 1.27e+26 | 221 3 | 21.2 Trillion |”
Citations
Cites
This paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.”
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 6×
“We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 9×
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38].”
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · cited 1×
“Finally, given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 / 2 (Robbins and Monro, 1951) (and may attain the bound).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 17×
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
InfluentialRobust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Cited by
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
“Empirical trends show this behavior improving with model scale, dataset size, and total training compute [56, 10, 21, 51].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
“Large language models (LLMs) Over the past few years, LLMs have shown impressive performance on natural language processing (NLP) tasks [12, 14, 15, 30, 69, 70, 73, 89, 91, 99].”
BackgroundLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
“Large language models (LLMs) Over the past few years, LLMs have shown impressive performance on natural language processing (NLP) tasks [12, 14, 15, 30, 69, 70, 73, 89, 91, 99].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“, 2021), Chinchilla (Hoffmann et al., 2022), Megatron–Turing NLG (Smith et al.”
BackgroundMethodResultInfluentialFoundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
“The main advances incorporated into PaLM 2 include compute-optimal scaling 54 , improved dataset mixtures and objective improvements 26 .”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
“GPT3 [55] Casual decoder 175B Pre Layer Norm Learned GeLU X 96 96 12288 2048 PanGU- α [73] Casual decoder 207B Pre Layer Norm Learned GeLU X 64 128 16384 1024 OPT [79] Casual decoder 175B Pre Layer Norm Learned ReLU X 96 96 12288 2048 PaLM [56] Casual decoder 540B Pre Layer Norm RoPE SwiGLU × 118 48 18432 2048 BLOOM […”
InfluentialDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
BackgroundAutonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Simulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937
(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
“Chinchilla : Chinchilla is a language model that has 70 billion parameters and has been pre-trained on 1.4 trillion tokens [78].”
BackgroundInfluentialCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
“LLMs generally follow scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) where performance increases with the size of the model and training data.”
BackgroundFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“• The Flamingo-9B model builds on top of a 7B frozen language model from [42].”
BackgroundMethodInfluentialSelf-citationStructured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771
“Large language models (LLMs) such as GPT-3/4 26,27 , PaLM 28 , Megatron 29 ,LLaMA1/2 30,31 ,OPT 32 ,Gopher 33 ,andFLAN 34 havebeenshown to have remarkable ability to leverage semantic information between tokens in natural language sequences of varying length.”
BackgroundThe Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., De Almeida, B. P., Sirelkhatim, H., et al. (2023) · doi:10.1101/2023.01.11.523679 · Cited by 680
“We extended the training duration of the 250 and 500 million parameter models to encompass 1 trillion tokens, aligning with recent recommendations in the literature [46] (Fig.”
MethodReproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673
“In particular, large models have led to substantial advances in speech [56], language [8, 17, 28, 57], vision [38, 84], and multi-modal language-vision settings [33,54,55,59,62].”
BackgroundMethodInfluentialNucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
“We extended the training duration of the 250M-and 500M-parameter models to encompass 1 trillion tokens, aligning with recent recommendations in the literature 45 (Fig.”
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“While it has been suggested that the model performance depends not only on model size but also on the volume of training data [220], the question of the right model size required to achieve the desired performance remains unclear.”
BackgroundCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
“LLMs generally follow scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) where performance increases with the size of the model and training data.”
BackgroundSequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
The Computational Limits of Deep Learning
Thompson, N. C., Greenewald, K., Lee, K., & Manso, G. F. (2020) · arXiv · doi:10.48550/arxiv.2007.05558 · Cited by 538
“Also, as we would expect given our discussion above, we see disagreements when different within-model scaling results are reported and conflict (significantly) in their recommendations [54, 118].”
ResultCode Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“Moreover, the Chinchilla scaling laws (Hoffmann et al., 2022) indicate that larger models would benefit more from training on more tokens.”
BackgroundMethodInfluentialLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
“While it has been suggested that the model performance depends not only on model size but also on the volume of training data [220], the question of the right model size required to achieve the desired performance remains unclear.”
BackgroundThe Computational Limits of Deep Learning
Thompson, N., Greenewald, K., Lee, K., & Manso, G. F. (2023) · Ninth Computing within Limits 2023 · doi:10.21428/bf6fb269.1f033948 · Cited by 495
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
“The AI models covered by this contribution are often referred to as ‘foundation models’ [36], ‘large language models’ (LLMs) [37] or ‘large generative models’ (LGAIMs–the term adopted in this article) [38].”
BackgroundLarge language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 370
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024) · IEEE Micro · doi:10.1109/mm.2024.3373763 · Cited by 369
“In this work, we re-examined this trend by studying more recent data, with a particular focus on hardware used to train AI models, as well as the characteristics of the computations used for training/serving them.”
MethodCompute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363
“The study of scaling laws [3, 9, 10, 11, 12, 13] relate these compute trends to model performance, and are actively used by large corporations to inform their training resource requirements [2, 14].”
BackgroundMethodPrecise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.99 · Cited by 362
Precise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2022) · arXiv · doi:10.48550/arxiv.2212.10496 · Cited by 362
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02272-z · Cited by 439
“It is possible that, in earlier phases of training, OpenFold has already recovered aspects of secondary structure not recognized by DSSP on account of imprecise atom positioning.”
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.507 · Cited by 316
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundGenerative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Machine learning for functional protein design
Notin, P., Rollins, N., Gal, Y., Sander, C., & Marks, D. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 270
“If trends observed in natural language processing 142 hold for proteins, leveraging larger datasets of protein sequences during training will yield improved generative models and, in turn, increase design quality.”
A foundation model for the Earth system
Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Allen, A., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J. A., Dong, H., et al. (2025) · Nature · doi:10.1038/s41586-025-09005-y · Cited by 267
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
“The 1.3-billion parameter models were trained for 30 billion tokens, while the 4-billion parameter LLMs received 100 billion tokens; both setups were consistent with compute-optimal scaling laws 29 .”
InfluentialLa nueva realidad de la educación ante los avances de la inteligencia artificial generativa
García Peñalvo, F. J., Llorens-Largo, F., & Vidal, J. (2023) · RIED-Revista Iberoamericana de Educación a Distancia · doi:10.5944/ried.27.1.37716 · Cited by 238
AI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233
“It is worth mentioning that the lack of awareness of covert racism also manifests during evaluation, where it is common to test language models for overt racism but not for covert racism 21,63,79,80 .”
Jablonka, K. M., Ai, Q., Al-Feghali, A., Badhwar, S., Bocarsly, J. D., Bran, A. M., Bringuier, S., Brinson, L. C., Choudhary, K., Circi, D., et al. (2023) · Digital Discovery · doi:10.1039/d3dd00113j · Cited by 218
Generative Artificial Intelligence in Education: Advancing Adaptive and Personalized Learning
Guettala, M., Bourekkache, S., Kazar, O., & Harous, S. (2024) · Acta Informatica Pragensia · doi:10.18267/j.aip.235 · Cited by 218
What large language models know and what people think they know
Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., & Smyth, P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00976-7 · Cited by 209
“Overall, there has been little research examining user confidence in LLM output.”
Larger and more instructable language models become less reliable
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024) · Nature · doi:10.1038/s41586-024-07930-y · Cited by 203
“FLOPs information usually captures both data and parameter count if models are well dimensioned 40 .”
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
“In 2022, a Cambrian explosion of natural language processing (NLP) models fl ooded the machine learning fi eld, from OpenAI ’ s GPT3 (1) to Google ’ s PALM (2), Gopher (3) and Chinchilla (4).”
BackgroundAI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
“…are largely driven by scaling laws , i.e., as these models grow in size—both in terms of their parameter count, and the data they are trained on—they tend to exhibit emergent abilities that are not present in small models [66], [67], [68], such as in-context learning, reasoning, and generalization.”
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169
Self-supervised learning for human activity recognition using 700,000 person-days of wearable data
Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01062-3 · Cited by 166
“The test performance for the AoT stayed at the random chance level, and the test performance for permutation dropped roughly 10 percentage points without weighted sampling.”
Large language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161
Choudhury, A., & Chaudhry, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/56764 · Cited by 159
“Although LLMs have shown promise in medical assessments [6-10], scientific writing, eHealth care, and patient classification [11-13], their integration marks a shift in paradigm introducing new AI complexities [14-17].”
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
Is GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144
“Following the established practice in language model training [33], we use a learning rate warm-up of 1000 steps and a cosine learning rate decay until a global minimum learning rate of 1e-7 across 20k steps.”
MethodCan language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142
How Secure is Code Generated by ChatGPT?
Khoury, R., Avila, A. R., Brunelle, J., & Camara, B. M. (2023) · 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc53992.2023.10394237 · Cited by 141
“This has been possible specially by means of increasing the model size, the training data and the model complexity [1].”
MethodThe Hydra Effect: Emergent Self-repair in Language Model Computations
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., & Legg, S. (2023) · arXiv · doi:10.48550/arxiv.2307.15771 · Cited by 140
“For more details on the Transformer architecture in language modelling and the specific details of Chinchilla language models see (Hoffmann et al., 2022; Phuong and Hutter, 2022).”
MethodPoisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138
“For example, the scaling laws observed in the recent Chinchilla [36] language model indicate that training a compute-optimal 500 billion parameter model would require 11 trillion tokens of training data—over 10 × more data than is currently used to train models of this size [23].”
MethodPoisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2302.10149 · Cited by 138
“For example, the scaling laws observed in the recent Chinchilla [36] language model indicate that training a compute-optimal 500 billion parameter model would require 11 trillion tokens of training data—over 10 × more data than is currently used to train models of this size [23].”
MethodBrinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132
“Workers who invest in related skills may outperform workers who do not, potentially accelerating the adoption of LLMs and further increasing the value of knowing how to collaborate with them.”
BackgroundA Theory for Emergence of Complex Skills in Language Models
Arora, S., & Goyal, A. (2023) · arXiv · doi:10.48550/arxiv.2307.15936 · Cited by 129
“For Chinchilla models Hoffmann et al. [2022] the law is as follows (precise constants of 2 hold only for the specific architecture and training strategy —even the constant A depends upon the tokenization): This description of macro behavior using two basic parameters —reminiscent of 2nd Law of…”
MethodNeural scaling of deep chemical models
Frey, N. C., Soklaski, R., Axelrod, S., Samsi, S., Gómez-Bombarelli, R., Coley, C. W., & Gadepally, V. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00740-3 · Cited by 125
Sai, S., Yashvardhan, U., Chamola, V., & Sikdar, B. (2024) · IEEE Access · doi:10.1109/access.2024.3385107 · Cited by 125
“LLMs like OpenAI ChatGPT, DeepMind Chinchilla AI [55], Meta LLaMA etc. have the ability to articulate their thought process, which enables them to probe in and translate the functionality of various technical files, including source code, configuration files etc., in plain language.”
MethodLanguage Models and Cognitive Automation for Economic Research
Korinek, A. (2023) · doi:10.3386/w30957 · Cited by 129
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Furuta, H., Lee, K.-H., Nachum, O., Matsuo, Y., Faust, A., Gu, S. S., & Gur, I. (2023) · arXiv · doi:10.48550/arxiv.2305.11854 · Cited by 115
“…could improve the data and parameter efficiency to achieve superior performance in offline regime, which is realized by the problem simplification of web navigation in order to leverage temporal-local visual perception and instruction-finetuned LLMs as strong inductive bias on web environments.”
BackgroundDeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113
“Following [35], [36], the training FLOPs can be estimated as 5.2 ZFLOPs, resulting in up to 18 days on 128 TESLA V100-32 GB GPUs.”
MethodInfluentialLearning the natural history of human disease with generative transformers
Shmatko, A., Jung, A. W., Gaurav, K., Brunak, S., Mortensen, L. H., Birney, E., Fitzgerald, T., & Gerstung, M. (2025) · Nature · doi:10.1038/s41586-025-09529-3 · Cited by 110
“A systematic screen of architecture hyperparameters (embedding dimensionality, number of layers, heads) confirms the reported empirical scaling laws 26 which state that model performance increases with the number of data points and, up to a limit defined by the available data, as the number of…”
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., & Khot, T. (2023) · arXiv · doi:10.48550/arxiv.2305.17306 · Cited by 109
“On GSM8k, it is worse (presumably because it is not trained on code) but close and much better than other open-sourced models (presumably because it is trained to Chinchilla-optimal Hoffmann et al., 2022).”
BackgroundSingh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.20879 · Cited by 108
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“However, as shown in Table 4, early works (Hoffmann et al., 2022; Kaplan et al., 2020) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws.”
BackgroundMethodInfluentialScalable Extraction of Training Data from (Production) Language Models
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., & Lee, K. (2023) · arXiv · doi:10.48550/arxiv.2311.17035 · Cited by 96
ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
Wake, N., Kanehira, A., Sasabuchi, K., Takamatsu, J., & Ikeuchi, K. (2023) · IEEE Access · doi:10.1109/access.2023.3310935 · Cited by 96
“The ability to perform few-shot learning is considered a result of increased model sizes [1] and extended training on large datasets [55].”
BackgroundARB: Advanced Reasoning Benchmark for Large Language Models
Sawada, T., Paleka, D., Havrilla, A., Tadepalli, P., Vidas, P., Kranias, A., Nay, J. J., Gupta, K., & Komatsuzaki, A. (2023) · arXiv · doi:10.48550/arxiv.2307.13692 · Cited by 92
“, 2022], and Chinchilla [Hoffmann et al., 2022] have shown increasing performance across a wide variety of natural language tasks ranging from translation to reasoning [Bubeck et al.”
BackgroundAisha: A Custom AI Library Chatbot Using the ChatGPT API
Lappalainen, Y., & Narayanan, N. (2023) · Journal of Web Librarianship · doi:10.1080/19322909.2023.2221477 · Cited by 89
“…the introduction of ChatGPT and many other LLM such as LaMDA (https://blog.google/technology/ai/lamda), AlexaTM (Soltan et al., 2022), Chinchilla (Hoffmann et al., 2022), PaLM (Chowdhery et al., 2022), PaLM 2 (https://ai.google/discover/palm2), Falcon (Technology Innovation Institute, 2023),…”
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
“…struggle to perform even simple tasks [98, 106, 108, 130, 163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific prestige [35, 67],…”
BackgroundWill we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79
“Large language models (LLMs) have demonstrated strong performance on a variety of reasoning tasks (Brown et al., 2020; Hoffmann et al., 2022; Chowd-hery et al., 2022, inter alia ).”
BackgroundA Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Voronov, A., Wolf, L., & Ryabinin, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.375 · Cited by 77
Understanding Emergent Abilities of Language Models from the Loss Perspective
Du, Z., Zeng, A., Dong, Y., & Tang, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1683 · Cited by 76
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
“[71] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Ja…”
BackgroundNovel machine learning approaches revolutionize protein knowledge
Bordin, N., Dallago, C., Heinzinger, M., Kim, S., Littmann, M., Rauer, C., Steinegger, M., Rost, B., & Orengo, C. (2023) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2022.11.001 · Cited by 72
“Orthogonal to such architectural improvements, recent research highlights the importance of hyperparameter optimization [120] which goes away from the constant increase in model size and rather suggests to train ‘smaller’ models (still, those models have billions of parameters) on more samples.”
BackgroundThe BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
“4 trillion tokens (Hoffmann et al., 2022) and dataset creation pipelines representing a significant part of large language model projects.”
BackgroundGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
“While it has been suggested that the model performance depends not only on model size but also on the volume of training data [87], the question of a triple-bottom-line model size that at the same time achieves performance, environmental and organizational objectives, remains unclear.”
The Quantization Model of Neural Scaling
Michaud, E. J., Liu, Z., Girit, U., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13506 · Cited by 57
“In practice however, larger networks are more efficient learners [7], and one can trade off between parameters and data, whereas in our model parameters and data independently bottleneck the number of quanta that can be learned.”
BackgroundKnow Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 56
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
Metadata
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., & Sifre, L.
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133225605
- Cited by
- 932
- References
- 78
- Citations by year
- 2026: 242 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26
- FWCI
- 96.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.