Training Compute-Optimal Large Language Models
Open accessThe DOI matches its Crossref record; 54 of the 78 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Overview
CoChat Summary
TL;DR
The paper estimates compute-optimal scaling for transformer language models and finds that model size and training tokens should be increased in roughly equal proportions as compute grows. It validates this by training Chinchilla, a 70B model on 1.4T tokens, which outperforms much larger models such as Gopher (280B) under the same compute budget.
Key findings
- Across three fitting approaches, the scaling exponents for optimal parameters \(N_{opt} \propto C^a\) and tokens \(D_{opt} \propto C^b\) are near equal: Approach 1 gives \(a=0.50, b=0.50\); Approach 2 gives \(a=0.49, b=0.51\); Approach 3 gives \(a=0.46, b=0.54\), in contrast to Kaplan et al. (2020), which reported \(a=0.73, b=0.27\).
- Chinchilla (70B parameters, 1.4T tokens) uses the same compute budget as Gopher (280B, 300B tokens) and outperforms Gopher, GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream tasks.
- Chinchilla reaches 67.6% average 5-shot accuracy on MMLU, improving on Gopher by 7.6%; it also improves BIG-bench average accuracy to 65.1% versus 54.4% for Gopher, and achieves 77.4% on LAMBADA zero-shot versus 74.5% for Gopher.
- The paper concludes that current large language models are significantly undertrained and that a 175B model should be trained on over 4.2 trillion tokens, while a 280B Gopher-like model should be trained on 6.8 trillion tokens.
Methods
- The authors train over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, then fit scaling relationships using three approaches: fixing model sizes and varying training tokens, IsoFLOP profiles, and fitting a parametric loss function.
- They use the resulting compute-optimal frontier to train Chinchilla, a 70B parameter autoregressive transformer, on 1.4 trillion tokens using the same compute budget as Gopher, with architecture and training setup based on Gopher but with changes including AdamW, a modified SentencePiece tokenizer without NFKC normalization, and a different MassiveText subset distribution.
- Evaluation covers language modelling, reading comprehension, question answering, common sense, MMLU, BIG-bench, gender bias, and toxicity, largely following the evaluation setup of Rae et al. (2021).
Limitations
- The paper states that due to the cost of training large models, it has only two comparable large-scale training runs (Chinchilla and Gopher) and no additional tests at intermediate scales.
- It assumes the efficient computational frontier follows a power-law relationship, but observes concavity in \(\log N_{opt}\) at high compute budgets, suggesting it may still overestimate the optimal size of large models.
- All training runs for the analysis used less than one epoch of data, so future work may consider the multiple-epoch regime; the paper also notes that language modelling comparisons may be affected by train/test leakage because Chinchilla is trained on 4× more data than Gopher, and its bias/toxicity evaluations are not comprehensive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
22 Valid · 5 Not valid · 9 Not checked · 88 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Section D.4, we show a head-to-head comparison at 10 21 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)-using the model size we predict has a clear advantage.”
What the sources say
- Supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)
Section 6.1 (p. 15) gives Kaplan et al.’s recommended model size as a function of compute budget, so their analysis can supply a model size for a comparison at 10^21 FLOPs. The claimed advantage of the citing authors’ model size is presented as the citing authors’ own finding, not as a finding attributed to Kaplan et al. (LLM, conf=0.90)
boostedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 2 (Robbins and Monro, 1951) (and may attain the bound).”
What the sources say
- Could not verifycites Robbins and Monro, 1951 matched to “A Stochastic Approximation Method” (1951) sourceExperimental — accuracy not yet measured
The Summary (p. 1) and Convergence theorems (pp. 2–5) establish convergence of a stochastic approximation procedure, but the readable passages do not give a 1/√D lower bound for the convergence rate of stochastic first-order methods. I searched for the rate, lower bound, and efficiency; several formulas in our copy are undecoded, so the claimed bound cannot be verified from it. (LLM, conf=0.72)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.”
What the sources say
- Supportedcites Izacard and Grave, 2020 matched to “Distilling Knowledge from Reader to Retriever for Question Answering” (2020) source
Table 2 (4.2 Technical Details, p. 7) reports a 72.5 TriviaQA test exact-match score for the paper’s open-book retriever–reader system, and 4.3 Results (p. 7) describes it as outperforming the prior state of the art. The citation supports the open-book SOTA benchmark it identifies; the 7.9-point comparison with Chinchilla is the citing authors’ calculation, not a result reported in this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
What the sources say
- Supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)
The citation identifies Kaplan et al.’s approach and results, rather than asking that paper to assess the citing authors’ approach. Section 3 (p. 7) describes the models Kaplan et al. trained, and the Discussion (p. 18) states their scaling results. The comparison with the citing authors’ results is their own conclusion. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.”
What the sources say
- Partly supportedcites Kaplan et al. (2020) matched to “Scaling Laws for Neural Language Models” (2020)Experimental — accuracy not yet measured
Section 6.1 (p. 15) says that each compute budget has an associated optimal model size and that the authors fit this relationship with a power law. Our copy does not give a readable coefficient or a value at 10^21 FLOPs, so it does not verify the specific figure of 4.68 billion parameters. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
45 more claims · 2 with their cited sources checked
Create a free account to see all 124 claims and to verify the 2 references still open.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 34
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Initially, scaling efforts focused on training-time expansion through increases in model and dataset size [14,18].”
From A Survey of Large Language Models (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Touvron et al. (2023a) are forced to pull numbers from the PaLM and Chinchilla technical reports (Chowdhery et al., 2023; Hoffmann et al., 2022) directly for baseline comparison, with no option to rerun these models on Llama's evaluation implementations since PaLM and Chinchilla are not publicly available.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformers are widely known to under-perform with insufficient training compute compared to RNNs [53, 40].”
From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measured
22 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Training Compute-Optimal Large Language Models · p. 1
“Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark”
§ 4.2.1. Language modelling · p. 11
“| Chinchilla 5-shot | 67.6% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2.5. Common sense · p. 12
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks”
§ 4.2.7. Gender bias and toxicity · p. 13
“| PIQA | 81.8% | 81.8% | 81.0% | 82.0% | 90.1% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 175 Billion | 4.54e+24 | 4.3 Trillion | 1.26e+24 | 12.0 Trillion |”
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 67 Billion | 6.88e+23 | 1.7 Trillion | 1.71e+24 | 4.1 Trillion |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4. Optimal model scaling · p. 8
“Unless one has a compute budget of 10 26 FLOPs (over 250 the compute used to train Gopher )”
§ 3.4. Optimal model scaling · p. 8
“| 1 Trillion | 1.27e+26 | 221 3 | 21.2 Trillion |”
Citations
Cites
This paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.”
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 6×
“We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 9×
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38].”
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · cited 1×
“Finally, given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 / 2 (Robbins and Monro, 1951) (and may attain the bound).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 17×
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
InfluentialRobust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Cited by
Items 301–353 of 353
Mu, S., Jiang, Y.-R., Wu, S., Hong, S., Sha, T., Zhang, J., Zhu, J., Chen, Z., Wang, Z., & Lin, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818445
“Inspired by the success of scaling laws in large language models (LLMs) [8, 10], the scaling laws of recommendation models have recently been widely studied and have achieved significant success in real-world industrial recommendation systems [6, 9, 30, 33, 34, 38, 41].”
The Scaling Paradox in Human-AI CollaborationPreprint
Qi, A., & Wang, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7207559
Harnessing Photonics for Machine Intelligence [Feature]
Zhu, H., Ning, S., Zhou, H., Yin, Z., Chen, R. T., Gu, J., & Pan, D. Z. (2026) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2026.3688606 · Cited by 5
“Underpinned by empirical scaling laws [3, 4], this evolution has shifted the field from isolated algorithmic novelty to systematic scaling of parameters and data, a trajectory that demands vastly more compute.”
AzText: Curating Web-Scale Pretraining Data for a Low-Resource Language
Mahammadli, E., & Rustamov, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31319-5_23
Hassan, M. T., Ahmed, J., & Awais, M. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3839365 · Cited by 3
“As demonstrated by the seminal ”Chinchilla” scaling laws [20], model performance is driven as much by the number of training tokens as by parameter count.”
A Review of Human-AI Complementarities Across Multiple Dimensions of Organisational Complexity
Sankaran, G., Palomino, M. A., & Siestrup, G. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10080268
“A significant empirical finding is that although performance continues to grow with data, parameters, and compute [30], data wield a disproportionately large impact [37].”
Toward WAN-Aware LLM Training Across Heterogeneous, Geo-Distributed Sites
Luo, Z.-Y., Cai, J., Denmat, C. L., et al. (2026) · Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication · doi:10.1145/3789240.3828748
“Large language model (LLM) training is increasingly concentrated in well-provisioned datacenters with homogeneous accelerators, low-jitter interconnects, and carefully engineered collective communication [2, 4, 10].”
A Comprehensive Evaluation of the Phi Family of Small Language Models in Healthcare
Awasthi, R., Shree, S., Bhattad, A., Mishra, S., Singh, N., Khare, A., Edara, S. R., Ramachandran, S. P., Mahapatra, D., Cywinski, J. B., et al. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00108
“However, Hoffmann et al. [10] later demonstrated that many existing LLMs are significantly under-trained for their size, and that compute-optimal scaling favors smaller models trained on larger datasets.”
A Dual-Teacher Distilled MoE Agent for Complex Industrial Document Analysis
Zhang, E., Kang, Q., Tang, R., Ren, M., Li, F., & Fang, J. (2026) · Applied Sciences · doi:10.3390/app16168089
Goswami, S., Jayaraman, R. G., Banerjee, P. S., & Rao, A. C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7292749
Liu, Z., Hou, S., Chen, G., Jiao, H., Wu, S., Xie, L., Yin, W., Qing, Y., Liang, J., Guan, X., et al. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134105
Large language models in bioinformatics: a comprehensive survey
Meng, Z., Yang, Z., Zhu, M., & Xu, J. (2026) · Frontiers in Genetics · doi:10.3389/fgene.2026.1797863
“5 (Radford et al., 2018; Radford et al., 2019; Brown et al., 2020; Ouyang et al., 2022) have made significant improvements in terms of task versatility and multimodal processing with the increase of parameters and training data (Fedus et al., 2022; Hoffmann et al., 2022).”
Contrastive Learning with Growing Generated Representations for Inductive Knowledge Graph Embedding
Li, Q., Cui, L., Joty, S., Wang, D., Feng, S., Zhang, Y., & Qin, C. (2026) · Data Science and Engineering · doi:10.1007/s41019-026-00352-y
Moralles, C., Costa, L. A. L. F. D., Silva, E. P., Prado, G. L. E., Schardosim, T. D. C., Rigo, S. J., Kunst, R., Souza, V. C., & Roehrs, A. (2026) · Anais do XVIII Simpósio Brasileiro de Computação Ubíqua e Pervasiva (SBCUP 2026) · doi:10.5753/sbcup.2026.20678
“However, the authors in [Hoffmann et al. 2022] demonstrated that many large models are un-dertrained and Chinchilla (70B) matched GPT-3 (175B) with better training.”
Gans, Joshua. The Microeconomics of Artificial Intelligence
Kasy, M. (2026) · Journal of Economic Literature · doi:10.1257/jel.64.3.1058.r4
ASLoRA: An Adaptive Scaling Method for Rotational LoRA in Federated Large Language Model Fine-Tuning
Han, X., Wu, S., Chen, C., Tian, H., Liu, G., & Dou, W. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00028
(2026) · Journal of Economic Literature · doi:10.1257/jel.64.3.1058
Fan, R., Zhou, Z., Huang, D., Du, X., Lu, H., & Li, Y. (2026) · Geocarto International · doi:10.1080/10106049.2026.2724469
Large language models in contexts of uncertainty: the “digital habitus” framework
Olivieri, M. G., & Iavarone, L. (2026) · Quality & Quantity · doi:10.1007/s11135-026-03049-6
Collaborative Large Model Caching and Inference Offloading With Parameter Sharing in MEC
Zhou, L., Xia, Q., Xu, Z., Xu, X., Xu, Z., Luo, R., Sheng, T., & Zhou, H. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3728847
Generative model to improve the physical simulation of normalized seismic acceleration
Jacquet, G., Gatti, F., & Clouteau, D. (2026) · Seismica · doi:10.26443/seismica.v5i2.1775
“We had yet to tackle the problem of data-model inflation, as expressed in the Chinchilla paper (Hoffmann et al., 2022), but a series of successive research and development have led us to this final approach.”
General World Models from First PrinciplesPreprint
Zhu, J., Tan, H., Zhang, J., Zhao, M., Bao, F., & Zhang, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7412285 · Cited by 2
Landscape Measure of Deep Learning Models
Grabovoy, A. V. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700513
“…progress in obtaining theoretical estimates of the spectral norms of Hessian matrices for various architectures [10, 14] and empirical regularities [6, 8], strict theoretical connections between the spectral properties of the Hessian and the generalization ability were established only for limited…”
Saygili, H. (2026) · Karadeniz Fen Bilimleri Dergisi · doi:10.31466/kfbd.1869329
Local large language model‐guided optimization for prostate radiotherapy planning
Nusrat, H., Luo, B., Bagher‐Ebadian, H., Hall, R., Kim, J., Doemer, A., Adil, K., Elsheikh, M., Feldman, A. M., Movsas, B., et al. (2026) · Medical Physics · doi:10.1002/mp.70654 · Cited by 2
Leveraging Vision Foundation Models for Robust Scene Understanding and Real-World Deployment
Alkhadour, W. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0136-3.ch007
Yoon, Y., & Kim, K. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7451102
Identifying cohorts at elevated risk of cancers using generative modeling of patient health statesPreprint
Khan, A., Forster, D. T., Harsh, M., Zheng, C., Warner, E. T., Ritter, D., Chang, A. E. B., Wei, Q., Sorensen, T. K., Marks, D. S., et al. (2026) · medRxiv · doi:10.64898/2026.09.09.26362676
da Silva, S. M. (2026) · Nexus Science Review · doi:10.66104/gs6h3421
“These results support continued improvement while also implying rising engineering complexity and the possibility that marginal gains become expensive (Hoffmann et al., 2022; Kaplan et al., 2020).”
What is Attention Mechanism? A Comprehensive Survey of Attention Methods and Transformer Models
Shiri, F. M., Memar, F., & Parhizgar, M. (2026) · Machine Learning · doi:10.1007/s10994-026-07131-w
Xie, Q., Li, Y., Wang, T., Ou, S., Chen, Z., & Zhang, L. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11689468
Umgang mit Sprachmodellen (Large Language Models)
Siebertz, K. (2026) · Springer eBooks · doi:10.1007/978-3-662-74590-8_2
Ping, J., & Liu, Z. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c02058
Measuring Reproducibility in LLM Research: A Rubric and Cross-Topic Corpus Study
Chong, C. J., Hou, C., Liu, S., & Yao, Z. (2026) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3820002.3828584
Mendonça, P. C., Quintal, F., Figueiredo, M. A. T., & Mendonça, F. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101343
Scaling Frontier AI: Compute Growth and the Sequence of Physical and Economic Ceilings
Alam, A., & Alam, M. S. (2026) · IEEE Access · doi:10.1109/access.2026.3737414
“Existing AI scaling work [11], [12] prescribes how to allocate a fixed compute budget optimally, but says nothing about how fast the budget itself is growing or what sustains that growth.”
Salah, K. M., Mahmoud, S. A., & Ibrahim, M. (2026) · Intelligent Methods, Systems, and Applications (IMSA) · doi:10.1109/imsa70415.2026.11700363
The Nature of AGI: Automated Hybrid Programming
林, 添. (2026) · Embedded Technology and Intelligent Systems · doi:10.12677/etis.2026.31001
Energy-Efficient AI for Foundation Models: Algorithms, Hardware, and Data Center Infrastructure
Bhupathiraju, K., Matharoo, R. S., Mwangi, H. W., Dagli, N., Power, A., Onsare, M. O., Lin, R., & Koçak, T. (2026) · Computers · doi:10.3390/computers15100660
Sejdiu, N. P., & Sejdiu, S. (2026) · Applied Linguistics · doi:10.1093/applin/amag089
Overview: Trends in AI Processing Chips
MOTOMURA, M. (2026) · The Journal of the Institute of Electrical Engineers of Japan · doi:10.1541/ieejjournal.146.585
Building a Sleep Data Repository
Maghsoudi, A., Mohamed, S. D., Pyarajan, S., Studnicki, A., & Werner, J. K. (2026) · Sleep Medicine Clinics · doi:10.1016/j.jsmc.2026.08.018
Enhancing Autonomous Agent Accuracy Through Reflection-Based Reasoning
Pang, L., Cen, C., Yu, J., Huang, D., & Sun, X. (2026) · International Conference on Advanced Network Technologies and Applications (APAN) · doi:10.1109/apan70967.2026.11709799
“This concern is consistent with broader work on compute-optimal training and efficient large-scale language model systems [23], [24].”
Ramli, M. S. B. (2026) · Research Square · doi:10.21203/rs.3.rs-11225296/v1
Böcekçi, S. C., & Yıldız, K. (2026) · Cluster Computing · doi:10.1007/s10586-026-06630-8 · Cited by 5
How Much Model Is Enough? Model Sufficiency for Workflow Generation
Alidu, A., Ciavotta, M., & Paoli, F. D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37831-6_4
Towards Embodiment Scaling Laws in Robot Locomotion
Ai, B., Dai, L., Bohlinger, N., Li, D., Mu, T., Wu, Z., Fay, K., Christensen, H. I., Peters, J., & Su, H. (2025) · arXiv · doi:10.48550/arxiv.2505.05753
“Scaling dataset size and model size has improved generalization in vision [13, 50, 61, 71, 91, 93, 99, 104] and language [1, 3, 20, 21, 30, 31, 38, 46, 72, 101].”
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
End-to-End Test-Time Training for Long Context
Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., et al. (2025) · arXiv
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
“For much of the development of Large Language Models (LLMs), scaling meant investing more computation before deployment by training larger models on more data [55, 46].”
Solving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358
“, 2015; Hosseini et al., 2014). Prompting language models using scratchpad Nye et al. (2021) or chain-of-thought Wei et al.”
BackgroundMethodByte Latent Transformer: Patches Scale Better Than Tokens
Pagnoni, A., Pasunuru, R., Rodriguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, L., Weston, J., Zettlemoyer, L., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.09871
Metadata
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., & Sifre, L.
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133225605
- Cited by
- 932
- References
- 78
- Citations by year
- 2026: 242 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26
- FWCI
- 96.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.