Training Compute-Optimal Large Language Models
Open accessThe DOI matches its Crossref record; 54 of the 78 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Overview
CoChat Summary
TL;DR
The paper estimates compute-optimal scaling for transformer language models and finds that model size and training tokens should be increased in roughly equal proportions as compute grows. It validates this by training Chinchilla, a 70B model on 1.4T tokens, which outperforms much larger models such as Gopher (280B) under the same compute budget.
Key findings
- Across three fitting approaches, the scaling exponents for optimal parameters \(N_{opt} \propto C^a\) and tokens \(D_{opt} \propto C^b\) are near equal: Approach 1 gives \(a=0.50, b=0.50\); Approach 2 gives \(a=0.49, b=0.51\); Approach 3 gives \(a=0.46, b=0.54\), in contrast to Kaplan et al. (2020), which reported \(a=0.73, b=0.27\).
- Chinchilla (70B parameters, 1.4T tokens) uses the same compute budget as Gopher (280B, 300B tokens) and outperforms Gopher, GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream tasks.
- Chinchilla reaches 67.6% average 5-shot accuracy on MMLU, improving on Gopher by 7.6%; it also improves BIG-bench average accuracy to 65.1% versus 54.4% for Gopher, and achieves 77.4% on LAMBADA zero-shot versus 74.5% for Gopher.
- The paper concludes that current large language models are significantly undertrained and that a 175B model should be trained on over 4.2 trillion tokens, while a 280B Gopher-like model should be trained on 6.8 trillion tokens.
Methods
- The authors train over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, then fit scaling relationships using three approaches: fixing model sizes and varying training tokens, IsoFLOP profiles, and fitting a parametric loss function.
- They use the resulting compute-optimal frontier to train Chinchilla, a 70B parameter autoregressive transformer, on 1.4 trillion tokens using the same compute budget as Gopher, with architecture and training setup based on Gopher but with changes including AdamW, a modified SentencePiece tokenizer without NFKC normalization, and a different MassiveText subset distribution.
- Evaluation covers language modelling, reading comprehension, question answering, common sense, MMLU, BIG-bench, gender bias, and toxicity, largely following the evaluation setup of Rae et al. (2021).
Limitations
- The paper states that due to the cost of training large models, it has only two comparable large-scale training runs (Chinchilla and Gopher) and no additional tests at intermediate scales.
- It assumes the efficient computational frontier follows a power-law relationship, but observes concavity in \(\log N_{opt}\) at high compute budgets, suggesting it may still overestimate the optimal size of large models.
- All training runs for the analysis used less than one epoch of data, so future work may consider the multiple-epoch regime; the paper also notes that language modelling comparisons may be affected by train/test leakage because Chinchilla is trained on 4× more data than Gopher, and its bias/toxicity evaluations are not comprehensive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
22 Valid · 5 Not valid · 9 Not checked · 88 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Section D.4, we show a head-to-head comparison at 10 21 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)-using the model size we predict has a clear advantage.”
What the sources say
- Supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)
Section 6.1 (p. 15) gives Kaplan et al.’s recommended model size as a function of compute budget, so their analysis can supply a model size for a comparison at 10^21 FLOPs. The claimed advantage of the citing authors’ model size is presented as the citing authors’ own finding, not as a finding attributed to Kaplan et al. (LLM, conf=0.90)
boostedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 2 (Robbins and Monro, 1951) (and may attain the bound).”
What the sources say
- Could not verifycites Robbins and Monro, 1951 matched to “A Stochastic Approximation Method” (1951) sourceExperimental — accuracy not yet measured
The Summary (p. 1) and Convergence theorems (pp. 2–5) establish convergence of a stochastic approximation procedure, but the readable passages do not give a 1/√D lower bound for the convergence rate of stochastic first-order methods. I searched for the rate, lower bound, and efficiency; several formulas in our copy are undecoded, so the claimed bound cannot be verified from it. (LLM, conf=0.72)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.”
What the sources say
- Supportedcites Izacard and Grave, 2020 matched to “Distilling Knowledge from Reader to Retriever for Question Answering” (2020) source
Table 2 (4.2 Technical Details, p. 7) reports a 72.5 TriviaQA test exact-match score for the paper’s open-book retriever–reader system, and 4.3 Results (p. 7) describes it as outperforming the prior state of the art. The citation supports the open-book SOTA benchmark it identifies; the 7.9-point comparison with Chinchilla is the citing authors’ calculation, not a result reported in this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
What the sources say
- Supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)
The citation identifies Kaplan et al.’s approach and results, rather than asking that paper to assess the citing authors’ approach. Section 3 (p. 7) describes the models Kaplan et al. trained, and the Discussion (p. 18) states their scaling results. The comparison with the citing authors’ results is their own conclusion. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.”
What the sources say
- Partly supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)Experimental — accuracy not yet measured
Section 6.1 (p. 15) says that each compute budget has an associated optimal model size and that the authors fit this relationship with a power law. Our copy does not give a readable coefficient or a value at 10^21 FLOPs, so it does not verify the specific figure of 4.68 billion parameters. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
45 more claims · 2 with their cited sources checked
Create a free account to see all 124 claims and to verify the 2 references still open.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 34
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Initially, scaling efforts focused on training-time expansion through increases in model and dataset size [14,18].”
From A Survey of Large Language Models (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Touvron et al. (2023a) are forced to pull numbers from the PaLM and Chinchilla technical reports (Chowdhery et al., 2023; Hoffmann et al., 2022) directly for baseline comparison, with no option to rerun these models on Llama's evaluation implementations since PaLM and Chinchilla are not publicly available.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformers are widely known to under-perform with insufficient training compute compared to RNNs [53, 40].”
From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measured
22 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Training Compute-Optimal Large Language Models · p. 1
“Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark”
§ 4.2.1. Language modelling · p. 11
“| Chinchilla 5-shot | 67.6% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2.5. Common sense · p. 12
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks”
§ 4.2.7. Gender bias and toxicity · p. 13
“| PIQA | 81.8% | 81.8% | 81.0% | 82.0% | 90.1% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 175 Billion | 4.54e+24 | 4.3 Trillion | 1.26e+24 | 12.0 Trillion |”
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 67 Billion | 6.88e+23 | 1.7 Trillion | 1.71e+24 | 4.1 Trillion |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4. Optimal model scaling · p. 8
“Unless one has a compute budget of 10 26 FLOPs (over 250 the compute used to train Gopher )”
§ 3.4. Optimal model scaling · p. 8
“| 1 Trillion | 1.27e+26 | 221 3 | 21.2 Trillion |”
Citations
Cites
This paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.”
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 6×
“We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 9×
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38].”
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · cited 1×
“Finally, given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 / 2 (Robbins and Monro, 1951) (and may attain the bound).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 17×
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
InfluentialRobust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Cited by
Items 101–200 of 353
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
“4 trillion tokens (Hoffmann et al., 2022) and dataset creation pipelines representing a significant part of large language model projects.”
BackgroundGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
“While it has been suggested that the model performance depends not only on model size but also on the volume of training data [87], the question of a triple-bottom-line model size that at the same time achieves performance, environmental and organizational objectives, remains unclear.”
Language Models as Zero-Shot Trajectory Generators
Kwon, T., Palo, N. D., & Johns, E. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3410155 · Cited by 59
The Quantization Model of Neural Scaling
Michaud, E. J., Liu, Z., Girit, U., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13506 · Cited by 57
“In practice however, larger networks are more efficient learners [7], and one can trade off between parameters and data, whereas in our model parameters and data independently bottleneck the number of quanta that can be learned.”
BackgroundKnow Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 56
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 700
“Despite progress in natural language generation (NLG) tasks like question answering or abstractive summarisation (Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2022), there is little understanding of uncertainty in foundation models.”
BackgroundWu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
“Despite this, performance does not rely solely on size, but also on training data volume [34].”
BackgroundSTELA: a community-centred approach to norm elicitation for AI alignment
Bergman, S., Marchal, N., Mellor, J., Mohamed, S., Gabriel, I., & Isaac, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-56648-4 · Cited by 48
“Auto red-teaming was achieved by employing few-shot parametrized dialogue prompts on DeepMind’s pre-trained but non-instruction-tuned 70B Chinchilla model 73 to mimic human adversarial probing (see Appendix A.1 for full specification).”
MethodScaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47
Large language models at work in China’s labor market
Chen, Q., Ge, J., Xie, H., Xu, X., & Yang, Y. (2025) · China Economic Review · doi:10.1016/j.chieco.2025.102413 · Cited by 47
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Song, X., Gupta, A., Mohebbizadeh, K., Hu, S., & Singh, A. (2023) · arXiv · doi:10.48550/arxiv.2305.14693 · Cited by 41
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
“We did not systematically explore fine-tuning due to the substantially higher compute required to do so.”
Studying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40
“There are some recent studies on debiasing reasoning in humans that might suggest a more direct way of comparing what humans do with LLMs.”
Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
Reviriego, P., Conde, J., Merino-Gómez, E., Martínez, G., & Hernández, J. A. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100602 · Cited by 37
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.29358 · Cited by 36
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
The mystery of In-context learning: a comprehensive survey on interpretation and analysis
Zhou, Y., Li, J., Xiang, Y., Yan, H., Gui, L., & He, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.00237 · Cited by 33
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31
“, DeepMind’s Chinchilla [2]) or employing a structured access approach to model sharing (e.”
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
The rising costs of training frontier AI models
Cottier, B., Rahman, R., Fattorini, L., Maslej, N., Besiroglu, T., & Owen, D. (2024) · arXiv · doi:10.48550/arxiv.2405.21015 · Cited by 27
“Improving AI capabilities demand exponential increases in computing power, as evidenced by both economic analysis [2] and the discovery of empirical scaling laws, which show that model performance improves with more parameters and training data [3, 4].”
BackgroundNi, B., Glaser, B., & Taheri-Mousavi, S. M. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01768-2 · Cited by 23
“With such a growing dataset, we expect to systematically test the model performance with various combinations of model con fi gurations and training data sizes, in hope to understand the scaling law of AlloyGPT models, triggeremergent capabilities 90 and achieve optimal performances 91 .”
International Governance of Civilian AI: A Jurisdictional Certification Approach
Trager, R., Harack, B., Reuel, A., Carnegie, A., Heim, L., Ho, L., Kreps, S., Lall, R., Larter, O., hÉigeartaigh, S. Ó., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.15514 · Cited by 23
Inverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25
Nezhurina, M., Cipolina-Kun, L., Cherti, M., & Jitsev, J. (2024) · arXiv · doi:10.48550/arxiv.2406.02061 · Cited by 21
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19
“For example, Hoffmann et al. (2022) showed that in order to get the most out of scaling a language model, one must scale the amount of training data at the same rate as the number of parameters.”
BackgroundCaution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
“, 2022), Chinchilla (Hoffmann et al., 2022), OPT(Zhang et al.”
BackgroundLeveraging Large Language Models for Scalable and Explainable Cybersecurity Log Analysis
Palma, G., Cecchi, G., Caronna, M., & Rizzo, A. (2025) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp5030055 · Cited by 17
eP-ALM: Efficient Perceptual Augmentation of Language Models
Shukor, M., Dancette, C., & Cord, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02016 · Cited by 16
“This work has demonstrated good performance, showing that it is possible to devise very efficient approaches to adapt existing language models [33, 78].”
BackgroundLearning to (Learn at Test Time): RNNs with Expressive Hidden States
Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04620 · Cited by 18
“The Chinchilla paper empirically observed that the compute-optimal models following their recipe fall onto a line in the log-log plot of FLOPs vs. perplexity, as is often the case for scaling law experiments [34].”
Compact Proofs of Model Performance via Mechanistic Interpretability
Gross, J., Agrawal, R., Kwa, T., Ong, E., Yip, C. H., Gibson, A., Noubir, S., & Chan, L. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2523 · Cited by 15
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
PepBERT: Lightweight language models for bioactive peptide representation
Du, Z., Caragea, D., Guo, X., & Li, Y. (2026) · Future Foods · doi:10.1016/j.fufo.2026.100999 · Cited by 17
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12
(2023) · doi:10.18653/v1/2023.acl-short.38 · Cited by 12
Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp
Hong, R., Agnew, W., Kohno, T., & Morgenstern, J. (2024) · Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3689904.3694702 · Cited by 11
Large Language Models for Scientific Information Extraction: An Empirical Study for Virology
Shamsabadi, M., D’Souza, J., & Auer, S. (2024) · Findings of the Association for Computational Linguistics: EACL 2024 · doi:10.18653/v1/2024.findings-eacl.26 · Cited by 11
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 305
What if eye...? Computationally recreating vision evolution
Tiwary, K., Young, A., Tasneem, Z., Klinghoffer, T., Dave, A., Poggio, T., Nilsson, D.-E., Cheung, B., & Raskar, R. (2025) · Science Advances · doi:10.1126/sciadv.ady2888 · Cited by 10
“While AI systems focus on scaling neural, data, and computation capacity ( 30 , 31 , 74 – 77 ), our findings suggest that evolution of embodied intelligent behavior (biological or artificial) may require coordinated scaling of visual acuity and processing.”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.01825 · Cited by 10
“Hoffmann et al. (2022) refined the scaling laws in (Kaplan et al., 2020) and found the scaling laws for computation-optimal training.”
BackgroundScaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9
“Despite the impressive capabilities of large language models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Anil et al., 2023; Gemini Team et al., 2023), connecting them to the embodied world that we inhabit remains challenging.”
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
“Inspired by (Kaplan et al., 2020; Hoffmann et al., 2022), we also found that the language modeling loss of our model follows a power-law plus constant scaling relationship with the context length (Figure 1), suggesting: 1) our model continues to show gain in language modeling up to 32,768 tokens of…”
MethodDemocratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Shashidhar, S., Chinta, A., Sahai, V., Wang, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.608 · Cited by 9
Peykani, P., Emrouznejad, A., Ghanidel, S., Javadi-Sisi, I., & Mirjalili, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10546-2 · Cited by 16
Exploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · arXiv · doi:10.48550/arxiv.2210.12517 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
Deng, H., & Raffel, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.721 · Cited by 5
“Large language models (LLMs, Rae et al., 2021; Hoffmann et al., 2022; Scao et al., 2022; Touvron et al., 2023) are seeing widespread adoption thanks to the fact that they can perform many language tasks and generate coherent long-form text.”
BackgroundSONIC: Supersizing motion tracking for natural humanoid whole-body control
Luo, Z., Yuan, Y., Wang, T., Li, C., Castañeda, F., Chen, S., Cao, Z.-A., Li, J., Minor, D., Ben, Q., et al. (2026) · Science Robotics · doi:10.1126/scirobotics.aed4592 · Cited by 344
“These foundation models have shown a consistent pattern: scale unlocks emergent capabilities, generalization, and robustness that smaller models cannot achieve (Bommasani et al., 2021; Hoffmann et al., 2022; Kaplan et al., 2020; Wei et al., 2022).”
Peng, Q., Chai, Y., & Li, X. (2024) · arXiv · doi:10.48550/arxiv.2402.16694 · Cited by 88
“This finding underscores the persistence of scaling laws (Hoffmann et al., 2022) while maintaining a consistent landscape of capabilities across a wide range of linguistic structures.”
BackgroundCan We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
de Cerqueira, J. A. S., Agbese, M., Rousi, R., Xi, N., Hamari, J., & Abrahamsson, P. (2024) · arXiv · doi:10.48550/arxiv.2411.08881 · Cited by 4
“These models can perform tasks such as language translation, text summarization, question answering, and creative writing [18].”
A Study on Machine Learning-Based Cost Estimation Models for AI Training Data Construction
Ko, Y.-S., & Lee, B. G. (2026) · Applied Sciences · doi:10.3390/app16062891 · Cited by 4
Spectral attention for transformers: frequency-domain filtering of attention maps
Huang, Z., Wu, P., Chen, M., Li, Q., & Pan, M. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00599-5 · Cited by 8
Agent Technology for Agricultural Intelligence: Methodological Framework and Applications
Li, Y., Wang, J., Yuan, Z., & Zhang, H. (2026) · Electronics · doi:10.3390/electronics15081547 · Cited by 5
Yeleussinov, A., Buribayev, Z., Beisov, N., Kalzhanov, N. Y., Satymbekov, M., Akhatov, U. A., & Alimkulov, Y. (2026) · Applied Sciences · doi:10.3390/app16136777 · Cited by 4
Large Language Models for Ontology Engineering: A Systematic Literature Review
Li, J., Garijo, D., & Poveda‐Villalón, M. (2026) · Semantic Web · doi:10.1177/22104968261465514 · Cited by 41
“LLMs require substantial computational resources for access and fine-tuning, which restricts their scalability (Hoffmann et al., 2022; Treviso et al., 2023).”
Towards more Human-like Language Models based on Contextualizer Pretraining Strategy
Xiao, C., Hudson, G. T., & Al Moubayed, N. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.28 · Cited by 3
Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
Kozlowski, A. C. (2026) · Theory and Society · doi:10.1007/s11186-026-09685-z · Cited by 4
“Much of the recent progress in machine learning, as well as the field's confi - dence in future progress, is attributable to a phenomenon known as scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020).”
Bridging language and action: A survey of language-conditioned robot manipulation
Yao, X., Zhou, H., Mees, O., Meng, Y., Xiao, T. G., Bisk, Y., Oh, J., Johns, E., Shridhar, M., Shah, D., et al. (2026) · The International Journal of Robotics Research · doi:10.1177/02783649261468360 · Cited by 58
“Many researchers attempt to study the performance limit of PLMs by scaling the size of models and datasets, 114 e.g., the comparatively small 1.5B parameter GPT-2, 115 versus larger 175B GPT-3, 116 and”
Gatt, A. (2025) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.896 · Cited by 2
Wittlinger, S., Meerjansen, J., Wolf, F., Wiest, I. C., Ebert, M. P., Siegel, F., & Belle, S. (2026) · medRxiv · doi:10.64898/2026.05.04.26352392 · Cited by 1
CLIMB – Curriculum Learning for Infant-inspired Model Building
Martinez, R. D., McGovern, H., Goriely, Z., Davis, C., Caines, A., Buttery, P., & Beinborn, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.10 · Cited by 2
CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning
Xu, C., Zhu, S., & Viswanathan, V. (2026) · Nature Communications · doi:10.1038/s41467-026-70467-3 · Cited by 8
Better language models better model the N400, but not reading time
Michaelov, J. A., & Bergen, B. K. (2026) · Journal of Memory and Language · doi:10.1016/j.jml.2026.104762 · Cited by 4
Digital Twins for Coastal Flooding
Dawson, C., Cerrone, A. R., Zhao, J., et al. (2026) · Computing in Science & Engineering · doi:10.1109/mcse.2026.3683247 · Cited by 1
The Epistemic Costs of Super-Persuasive AI
Deller, A. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01106-4 · Cited by 2
Chen, C., Wang, Y., Guo, Y., Yang, H., Zhang, H., & Dong, Z. (2026) · Sustainability · doi:10.3390/su18104683 · Cited by 1
Performance Trade-Offs of Optimizing Small Language Models for E-Commerce
Licardo, J. T., Tanković, N., Osman, I., Lorencin, I., & Šegota, S. B. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10050155 · Cited by 5
Ma, Y., Xing, Y., Wu, D., Zhou, Y., Zi, Y., Wang, M., Deng, Y., & Pan, S. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351493 · Cited by 2
Data protection in the era of large language models: challenges and opportunities
Revesai, Z., & Chipfumbu, C. T. (2026) · AI and Ethics · doi:10.1007/s43681-026-01223-y · Cited by 5
“GPT-3 contains 175 billion parameters, while GPT-4 likely exceeds one trillion [29 ].”
Scaling transferable coarse-graining with mean force matching
Park, A., Chennakesavalu, S., & Rotskoff, G. M. (2026) · The Journal of Chemical Physics · doi:10.1063/5.0329526 · Cited by 4
“As is well-documented across many domains in the machine learning literature [21], increases in model size and training data lead to predictable performance improvements and enable compute-optimal resource allocation [22] due to the strong adherence to neural scaling laws.”
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 4
“Prior work has described power-law relationships between training loss and key scaling factors, including model parameters, dataset size, and training compute, suggesting that better performance can be achieved by increasing resources under a fixed optimization recipe [10,74].”
InfluentialTaricco, G. (2026) · Entropy · doi:10.3390/e28070811 · Cited by 1
Can large language models be a cardinality estimator? An empirical study
Liu, L., Wu, Y., Wang, Y., Chang, Z., Su, R., Wu, P., Chen, J., Jiang, F., Shi, R., Cui, B., et al. (2026) · The VLDB Journal · doi:10.1007/s00778-026-00975-7 · Cited by 1
“Moreover, analyses of zero-shot performance suggest that apparent zero-shot gains often arise from latent overlap between evaluation tasks and pretraining data [22, 87].”
Ahmadzadeh, S., Karthick, G., & Alsafi, T. (2026) · Future Internet · doi:10.3390/fi18080393 · Cited by 8
Peykani, P., Charles, V., Emrouznejad, A., Ghanidel, S., & Sabour, D. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10720-6 · Cited by 1
Exact sequence interpolation with transformers
Alcalde, A., Fantuzzi, G., & Zuazua, E. (2026) · Mathematical Foundations of Machine Learning · doi:10.1007/s44439-026-00005-y · Cited by 6
“This reformulation leads to a complexity that scales with input length n rather than output length m , a scaling also observed in large-scale models [40].”
Advances in reasoning by prompting large language models: A survey
Hong, R., Pang, X., & Zhang, C. (2026) · Cybernetics and Intelligence · doi:10.26599/cai.2024.9390004 · Cited by 11
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
“Scaling LLMs and RL. Scaling laws for language models show diminishing returns on the loss for the single step of predicting the next token (Kaplan et al., 2020; Hoffmann et al., 2022).”
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“…(3) designing more efficient/effective optimization algorithms to find better solutions or with more resilience to forgetting [33–36]; and (4) scaling the model size to enhance its expressivity, when the “right” choice of architecture, objective, and optimization algorithms are made [24, 37, 38].”
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · doi:10.21203/rs.3.rs-8919408/v1 · Cited by 1
Evaluating the learnability of single-cell large language models on multiple tasks
Yan, Y., Wang, X., & Song, D. (2026) · BMC Genomics · doi:10.1186/s12864-026-12975-6 · Cited by 1
BOND: Aligning LLMs with Best-of-N Distillation
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.14622 · Cited by 1
“BOND hinges on the idea of investing more resources during training to ensure that computational demands during inference remain low, a factor often overlooked in traditional scaling laws (Hoffmann et al., 2022).”
BackgroundResearch, C., :, Chan, A., Shalaby, A., Wettig, A., Sanger, A., Zhai, A., Ajay, A., Nair, A., Snell, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.24477 · Cited by 1
“Taking this a step further, recent models use a staged training approach, progressively filtering towards higher quality data [22, 69, 77].”
The economics of accuracy for medical reasoning with large language models
Bhattacharyya, K., & Kamabattula, S. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001182 · Cited by 1
“However, the dominant paradigm in artificial intelligence (AI) has held that larger models, trained on larger datasets, would yield superior performance—a perspective encapsulated in the neural scaling laws that governed model development [4,5].”
Ghost Mechanism: An Analytical Model of Abrupt Learning in Recurrent Networks
Dinç, F., Cirakman, E., Kurtkaya, B., Yüksekgönül, M., Jiang, Y., Schnitzer, M. J., & Tanaka, H. (2026) · Physical Review X · doi:10.1103/mjcl-lb4x · Cited by 8
Survey of Large Language Models: Exploring Options Beyond ChatGPT
Wadhwa, G., Hummel, S., Abbas, S. H., & Donner, M.-T. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-3420-0_29
Sustainable Computing and Green AI
Dahiya, O., & Gotter, P. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5540-5.ch001
Diving Deep Into the World of LLMs
Saleh, M. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3912-2.ch001
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135630 · Cited by 3
Architectural Laws And Derived Constraints Of Verification And Consequential AI ReliancePreprint
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135727
Tail Task Risk Minimization in Meta-Learning From Theoretical Advances to Practical Strategies
Lv, Y., Liang, D., Du, W., Shi, Z., Xie, Z., Wang, Q., & Wang, M. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3665061
“In training distributionally robust meta learners, estimating the convergence rate allows monitoring of the convergence and designing early stopping criteria to reach a desirable performance, reducing computational burdens [79].”
Metadata
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., & Sifre, L.
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133225605
- Cited by
- 932
- References
- 78
- Citations by year
- 2026: 242 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26
- FWCI
- 96.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.