Training Compute-Optimal Large Language Models
Open accessThe DOI matches its Crossref record; 53 of the 78 references it cites were found in a registry or an index; 26 of the 38 sources its claims cite were checked for support.
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Overview
CoChat Summary
TL;DR
The paper estimates compute-optimal scaling for transformer language models and finds that model size and training tokens should be increased in roughly equal proportions as compute grows. It validates this by training Chinchilla, a 70B model on 1.4T tokens, which outperforms much larger models such as Gopher (280B) under the same compute budget.
Key findings
- Across three fitting approaches, the scaling exponents for optimal parameters \(N_{opt} \propto C^a\) and tokens \(D_{opt} \propto C^b\) are near equal: Approach 1 gives \(a=0.50, b=0.50\); Approach 2 gives \(a=0.49, b=0.51\); Approach 3 gives \(a=0.46, b=0.54\), in contrast to Kaplan et al. (2020), which reported \(a=0.73, b=0.27\).
- Chinchilla (70B parameters, 1.4T tokens) uses the same compute budget as Gopher (280B, 300B tokens) and outperforms Gopher, GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream tasks.
- Chinchilla reaches 67.6% average 5-shot accuracy on MMLU, improving on Gopher by 7.6%; it also improves BIG-bench average accuracy to 65.1% versus 54.4% for Gopher, and achieves 77.4% on LAMBADA zero-shot versus 74.5% for Gopher.
- The paper concludes that current large language models are significantly undertrained and that a 175B model should be trained on over 4.2 trillion tokens, while a 280B Gopher-like model should be trained on 6.8 trillion tokens.
Methods
- The authors train over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, then fit scaling relationships using three approaches: fixing model sizes and varying training tokens, IsoFLOP profiles, and fitting a parametric loss function.
- They use the resulting compute-optimal frontier to train Chinchilla, a 70B parameter autoregressive transformer, on 1.4 trillion tokens using the same compute budget as Gopher, with architecture and training setup based on Gopher but with changes including AdamW, a modified SentencePiece tokenizer without NFKC normalization, and a different MassiveText subset distribution.
- Evaluation covers language modelling, reading comprehension, question answering, common sense, MMLU, BIG-bench, gender bias, and toxicity, largely following the evaluation setup of Rae et al. (2021).
Limitations
- The paper states that due to the cost of training large models, it has only two comparable large-scale training runs (Chinchilla and Gopher) and no additional tests at intermediate scales.
- It assumes the efficient computational frontier follows a power-law relationship, but observes concavity in \(\log N_{opt}\) at high compute budgets, suggesting it may still overestimate the optimal size of large models.
- All training runs for the analysis used less than one epoch of data, so future work may consider the multiple-epoch regime; the paper also notes that language modelling comparisons may be affected by train/test leakage because Chinchilla is trained on 4× more data than Gopher, and its bias/toxicity evaluations are not comprehensive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
22 Valid · 5 Not valid · 9 Not checked · 88 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Section D.4, we show a head-to-head comparison at 10 21 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)-using the model size we predict has a clear advantage.”
What the sources say
- Supportedcites Kaplan et al. (2020)
Section 6.1 (p. 15) gives Kaplan et al.’s recommended model size as a function of compute budget, so their analysis can supply a model size for a comparison at 10^21 FLOPs. The claimed advantage of the citing authors’ model size is presented as the citing authors’ own finding, not as a finding attributed to Kaplan et al. (LLM, conf=0.90)
boostedstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 2 (Robbins and Monro, 1951) (and may attain the bound).”
What the sources say
- Could not verifycites Robbins and Monro, 1951 sourceExperimental — accuracy not yet measured
The Summary (p. 1) and Convergence theorems (pp. 2–5) establish convergence of a stochastic approximation procedure, but the readable passages do not give a 1/√D lower bound for the convergence rate of stochastic first-order methods. I searched for the rate, lower bound, and efficiency; several formulas in our copy are undecoded, so the claimed bound cannot be verified from it. (LLM, conf=0.72)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.”
What the sources say
- Supportedcites Izacard and Grave, 2020 source
Table 2 (4.2 Technical Details, p. 7) reports a 72.5 TriviaQA test exact-match score for the paper’s open-book retriever–reader system, and 4.3 Results (p. 7) describes it as outperforming the prior state of the art. The citation supports the open-book SOTA benchmark it identifies; the 7.9-point comparison with Chinchilla is the citing authors’ calculation, not a result reported in this source. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
What the sources say
- Supportedcites Kaplan et al. (2020)
The citation identifies Kaplan et al.’s approach and results, rather than asking that paper to assess the citing authors’ approach. Section 3 (p. 7) describes the models Kaplan et al. trained, and the Discussion (p. 18) states their scaling results. The comparison with the citing authors’ results is their own conclusion. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.”
What the sources say
- Partly supportedcites Kaplan et al. (2020)Experimental — accuracy not yet measured
Section 6.1 (p. 15) says that each compute budget has an associated optimal model size and that the authors fit this relationship with a power law. Our copy does not give a readable coefficient or a value at 10^21 FLOPs, so it does not verify the specific figure of 4.68 billion parameters. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
45 more claims · 2 with their cited sources checked
Create a free account to see all 124 claims and to verify the 2 references still open.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 34
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Initially, scaling efforts focused on training-time expansion through increases in model and dataset size [14,18].”
From A Survey of Large Language Models (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Touvron et al. (2023a) are forced to pull numbers from the PaLM and Chinchilla technical reports (Chowdhery et al., 2023; Hoffmann et al., 2022) directly for baseline comparison, with no option to rerun these models on Llama's evaluation implementations since PaLM and Chinchilla are not publicly available.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformers are widely known to under-perform with insufficient training compute compared to RNNs [53, 40].”
From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Hoffmann et al., 2022
Experimental — accuracy not yet measured
22 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Training Compute-Optimal Large Language Models · p. 1
“Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark”
§ 4.2.1. Language modelling · p. 11
“| Chinchilla 5-shot | 67.6% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2.5. Common sense · p. 12
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks”
§ 4.2.7. Gender bias and toxicity · p. 13
“| PIQA | 81.8% | 81.8% | 81.0% | 82.0% | 90.1% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 175 Billion | 4.54e+24 | 4.3 Trillion | 1.26e+24 | 12.0 Trillion |”
§ D.3. Predicted compute optimal frontier for all three methods · p. 26
“| 67 Billion | 6.88e+23 | 1.7 Trillion | 1.71e+24 | 4.1 Trillion |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4. Optimal model scaling · p. 8
“Unless one has a compute budget of 10 26 FLOPs (over 250 the compute used to train Gopher )”
§ 3.4. Optimal model scaling · p. 8
“| 1 Trillion | 1.27e+26 | 221 3 | 21.2 Trillion |”
Citations
Cites
This paper’s reference list could not be split into single entries.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.”
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 6×
“We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 9×
“We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38].”
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · cited 1×
“Finally, given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 𝐷 1 / 2 (Robbins and Monro, 1951) (and may attain the bound).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 17×
“Our approach leads to considerably different results than that of Kaplan et al. (2020).”
InfluentialRobust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).”
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Cited by
Items 201–300 of 348
Power Couple? AI Growth and Renewable Energy InvestmentPreprint
Gui, L., & Dai, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6166290
Beyond Perplexity: A Multi-Faceted Analysis of a Novel Densely Connected Transformer
Santis, E. D., Martino, A., & Rizzi, A. (2026) · Applied Sciences · doi:10.3390/app16062721 · Cited by 1
Spadaccino, P., Lorenzo, P. D., Barbarossa, S., Tulino, A. M., & Llorca, J. (2026) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2026.3673194
“Specifcially, modern AI workloads executed on GPUs load large deep-learning models that often saturate VRAM [27], [28].”
No skin in the game: why agentic AI requires principal-agent governance
Prause, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01067-6 · Cited by 2
Teaching an Agent to Sketch One Part at a Time
Du, X., Xu, R., Yunis, D., Vinker, Y., & Shakhnarovich, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37032-7_14 · Cited by 3
Accelerating Personalization Signal Learning via Synthetic Data
Parveen, D., Kang, D., Paruchuri, A., Kayal, D., & Mallapragada, P. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_17
“For fine-tuning, we utilize LLMs with parameters ≤ 10 B, balancing performance with production deployment constraints such as inference latency, and GPU capacity ([12],[9]).”
Wang, Z., Li, Y., Fu, J., & Zhang, G. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_4
KFPT: Reliability and uncertainty filtered self-distillation for language model training
Yuce, M. K., & Amasyalı, M. F. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115880
Müller, B. (2022) · theses.fr (ABES) · doi:10.70675/6984dc63z4731z4eabzb3fazaf5d012fe143
Teodorescu, L. (2023) · theses.fr (ABES) · doi:10.70675/08e0bf33z987fz4ff6zbdd7zbf6cf87e81f0 · Cited by 3
Representation learning of multimodal and longitudinal data for renal transplantation monitoring
Milecki, L. (2024) · theses.fr (ABES) · doi:10.70675/8478bb39za27fz4524z8e86z5df6c7d2fc00 · Cited by 3
Improving Representations for Language Modeling
Godey, N. (2024) · theses.fr (ABES) · doi:10.70675/b6a15d0ez15bez43e5za967z16a069e0a8bb
Répondre aux questions visuelles à propos d'entités nommées
Lerner, P. J. (2023) · theses.fr (ABES) · doi:10.70675/78c96766zfc95z4da8zb67aze67555da6c75 · Cited by 1
Diverse and efficient ensembling of deep networks
Ramé, A. (2023) · theses.fr (ABES) · doi:10.70675/fa295dcfz969ez4448z9a30z7c7bbfabb740
Spoken Language Modeling from Raw Audio
Nguyen, T. A. (2024) · theses.fr (ABES) · doi:10.70675/e81cb131z2de8z4c0fz86f5zd983c6cb8744 · Cited by 1
Language models for document understanding
Douzon, T. (2023) · theses.fr (ABES) · doi:10.70675/66ce60f8zcccaz476eza4d6zf801a2256b86 · Cited by 23
Harnessing symmetries for modern deep learning challenges : a path-lifting perspective
Gonon, A. (2024) · theses.fr (ABES) · doi:10.70675/c0dd6b18z746ez4cfezae65zaba0b12d7641 · Cited by 5
Foundation Vision-Language models
Laurençon, H. (2025) · theses.fr (ABES) · doi:10.70675/ec0311a0zb6a4z4249zb9fbz01bf51a02c7c · Cited by 1
Peuplement semi-automatisé de bases de connaissances à partir de documents non structurés.
Prieur, M. (2025) · theses.fr (ABES) · doi:10.70675/6c995326zb3b9z422dz9e5azbbabf8c75862
Riabi, A. (2025) · theses.fr (ABES) · doi:10.70675/5a649ef5za082z43d8z906azd8051be91d15 · Cited by 4
Artificial Intelligence (AI) Main Concepts: A Glimpse inside “A.I. Black Box”
Mantzaris, D. H., & Karampinis, E. (2026) · Springer eBooks · doi:10.1007/978-3-032-15614-3_1 · Cited by 1
Research on the Tiered Pricing Model for Intelligent Computing Power Trading
梁, 秉. (2026) · Software Engineering and Applications · doi:10.12677/sea.2026.152014
Adaptive Self-Prompting in Agentic LLM Frameworks for Code Fault Detection
Muhtadi, M., Mahmoud, Q. H., & Azim, A. (2026) · Software · doi:10.3390/software5020016
“In contrast, larger models (e.g., 70B+) incur substantially higher computational and memory costs, with diminishing efficiency returns relative to their scaling, making them less practical for real-time or resource-constrained deployment scenarios [34].”
Muñoz-Babiano, F., Lamo, P., & Alonso, R. S. (2026) · ADCAIJ ADVANCES IN DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE JOURNAL · doi:10.14201/adcaij.32926
Vector Quantization-Based Watermarking for Autoregressive Generated Images
Lai, Q., & Borş, A. G. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464962
Optimizing speculative decoding via dynamic multi-path and dual-stream networks
Yang, Y., Yang, T., Gao, S. S., Ning, J. H., & Kong, L. Z. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109014 · Cited by 1
Wang, P., Shi, D., Li, M., & Aguilar, J. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02760-y · Cited by 1
Mantshimuli, L., & Mwamba, J. W. M. (2026) · Journal of Risk and Financial Management · doi:10.3390/jrfm19050320 · Cited by 1
“Additionally, optimal training practices, such as balancing the model size with the training data volume, have been shown to further enhance model capabilities while reducing the computational costs Hoffmann et al. (2022).”
Chen, D., Gong, Y., Chen, J., & He, R. (2026) · Energy and Buildings · doi:10.1016/j.enbuild.2026.117578 · Cited by 2
Toward a Unified Vision-Language Generative Model
Wang, P. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1014640 · Cited by 4
Liu, K., Xiong, H., Zhang, J., & Peng, M. (2026) · Electronics · doi:10.3390/electronics15102078 · Cited by 1
DEVELOPMENT OF A DECISION SUPPORT SYSTEM USING ARTIFICIAL NEURAL NETWORK AND GENETIC ALGORITHM
Rybak, O. (2026) · Системи управління навігації та зв’язку Збірник наукових праць · doi:10.26906/sunz.2026.2.176
Quantization-Induced Degradation Scaling Laws: A Unified Mathematical Framework
Hawwari, M., Yilmazer, Y., Inan, K., Turkmen, G., Cetintas, Z., Lee, J., & Baburoglu, I. (2026) · doi:10.65737/airjet2026717
Dynamic Normativity: Learning Human Preferences
Corrêa, N. K. (2026) · The International library of ethics, law and technology · doi:10.1007/978-3-032-22150-6_5
Worldwide AI Ethics, Principles, and Blank Spots
Corrêa, N. K. (2026) · The International library of ethics, law and technology · doi:10.1007/978-3-032-22150-6_1
Data and Knowledge Dual-Driven Artificial Intelligence
Wang, H., Wang, M., Liu, L., & Wu, J. (2026) · Springer eBooks · doi:10.1007/978-981-95-5585-7_4
The Thermodynamic Costs of Simple Linear Regression
D’Ambrosia, S. H., Daniels, S., DeWeese, M. R., & Sahai, A. (2026) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2026.3732971 · Cited by 1
“Empirical scaling laws [63], [64] have become tools for deep learning practitioners to set the best model and dataset size for a fixed training compute budget.”
Cross-sectional analysis of large language models in current natural language generation challenges
Maestre, M. M., Martínez-Murillo, I., Martínez, A. M., & Lloret, E. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104922 · Cited by 1
Ge, Y., Gu, J., & Huang, C. (2026) · Data · doi:10.3390/data11060126
Yang, H., Guo, J., Wang, Y., & Zhang, X. (2026) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-026-00921-6
“The main contributions of this study are summarized as follows: ( foundation, models such as BART [15] and Chinchilla [16] have achieved significant improvements in language understanding and generation tasks, demonstrating strong capabilities in text completion, summarization, and reasoning.”
ILL: A Lightweight Large Language Model for Legal and Courtroom Assistance
Lin, Z., Liang, J., Zhao, X., Zhu, Z., Hu, W., & Liu, X. (2026) · Electronics · doi:10.3390/electronics15112401
Training transformers for mesh-based simulations
Garnier, P., Lannelongue, V., Viquerat, J., & Hachem, E. (2026) · Neural Computing and Applications · doi:10.1007/s00521-025-11731-3 · Cited by 6
“For example, we add a x , a y , a z to each node from an aneurysm mesh.”
InfluentialEnhancing feature diversity with weak teacher directed parameters perturbation
Li, H., Pu, Y., Miao, Z., Wang, J., & Li, Y. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02407-6
Unifying Data, Memory, and Compute Efficiency in LLM Training: A Survey
Schmidt, V., Nguyen, H. H., Jung, C., Salehi, S., & Schmeink, A. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3702625
“This picture is later revised by works such as Hoffmann et al. [29], and Muennighoff et al. [30], which argue for a more balanced, data-rich notion of compute-optimality.”
InfluentialSustainability via Large Language Model Right-Sizing
Haase, J., Klessascheck, F., Mendling, J., & Pokutta, S. (2026) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2026.3697261 · Cited by 4
“For example, the Chin-chilla model (70B parameters) outperformed the larger Gopher model (280B) by optimizing the balance between model size and training dataset size [33], [34].”
Security, Privacy, and Ethical Challenges in Large Language Models: A Review
Yogi, B., Ghosh, N., & Sahid, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10680-x
Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems
Gravara, M., Stanisic, A., & Nastic, S. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00049 · Cited by 3
“Scaling laws establish that pretraining loss decreases as a power law with respect to model size, dataset size, and compute [4], [8].”
InfluentialOrquestração de dispositivos IoT em cenários complexos com interação humana baseada em LLMs
Perin, A. O., Pinto, M. V. A., Macêdo, R. J. D. A., Santos, B. P., Peixoto, M. L. M., Figueiredo, G. B., & Prazeres, C. V. S. (2026) · doi:10.5753/sbrc.2026.19352
Elucidating the Design Space of Generative Models for Single-Cell Perturbation PredictionPreprint
Bhattacharya, S., Gensbigler, C., Karim, S., & Lees, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.15.732063
“Autoregressive models have demonstrated predictable scaling laws on language data (14), an appealing property as virtual cell models scale up, and they operate naturally over discrete tokens.”
Nagre, P., Rewatkar, R. M., Dube, N., et al. (2026) · doi:10.1109/icima68728.2026.11564766
“The latest Large Language Models (LLMs), such as GPT-4 or Llama-2-70B, demand server-grade GPUs with hundreds of gigabytes of VRAM, which is much beyond the hardware capabilities of consumer hardware [4], [13].”
Development of a domain-tailored on-premise dialogue system for accelerated materials analysis
Goto, S., Kamiyama, M., Sudo, E., & Kimura, H. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12254-1
Ra C : Robot Learning for Long-Horizon Tasks by Scaling R ecovery a nd C orrection
Hu, Z., Wu, R., Enock, N., Li, J., Kadakia, R., Erickson, Z., & Kumar, A. (2026) · IEEE Transactions on Robotics · doi:10.1109/tro.2026.3706552 · Cited by 56
“We formalize this notion of recovery and correction and develop an approach to collect data naturally rich in these behaviors.”
Reasoning models, test-time compute, self refinement
Zsolt, J. L. (2026) · IEEE International Conference and Workshop Óbuda on Electrical and Power Engineering (CANDO-EPE) · doi:10.1109/cando-epe71091.2026.11569472
“Generally, increasing these factors leads to a proportional improvement in model performance, as formalized by the Kaplan [7] and Chinchilla [8] scaling laws.”
Public perceptions of AI energy consumption: Awareness, responsibility attribution, and strategies
Klesel, M., & Messer, U. (2026) · Electronic Markets · doi:10.1007/s12525-026-00908-w · Cited by 2
Benchmarking Multimodal Mathematical Reasoning: Prompt Effects, Modality Gaps, and Failure Modes
Görer, G., Osipenko, M., & Knispel, T. (2026) · Metrics · doi:10.3390/metrics3030011
A Review on Systematically Enhancing the Application Capabilities of Large Language Models
Zong, J., Yuan, L., Zhang, J., Gao, X., & Hou, Z. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-7668-5_45
FASST-LLM: Implementing Service Scanning Tools Automatically with LLM
Qin, L., Luo, Y., Li, C., Wang, Z., & Yang, J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571366
“Large Language Models (LLMs) have shown remarkable emergent abilities as they scale [20] [21], including instruction following [22], in-context learning [23], and Chain-of-Thought (CoT) reasoning [24].”
Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputation
Kim, G.-W., Kim, D., Moon, J., Liu, H., Khan, T., Iyer, A. P., Kim, D., & Akella, A. (2026) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps65963.2026.00071 · Cited by 5
“Recent studies have enabled reasoning on graph-structured data [14]–[18] by enhancing large language models (LLMs) [19]–[22] with GNNs to effectively capture intricate structural information.”
ExMe: Keep Improving With Extrapolation and Merging
Lin, Y., Xu, B., LI, Y., & Gao, Y. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3708509 · Cited by 3
“This phase often involves supervised post-training, such as fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) [5], [6], both of which demand high-quality data for downstream tasks and significant computational resources [7].”
F-AVSS: A Fuzzy Layer-Importance Metric for Efficient Large Language Model Compression
Song, Z., Kang, X., & Kang, Z. (2026) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2026.3708997
“Visualizations of membership surfaces and rule-activation heat maps further highlight the interpretability of the pruning decisions [6], [7], [8], [9], [10], [11], [12].”
Peer-to-Peer Federated Learning: A Comprehensive Survey
Allen, A., Mylonas, A., Vidalis, S., & Pitropakis, N. (2026) · AI · doi:10.3390/ai7070243 · Cited by 2
Embedding Neural Scaling Laws Into Optimization Dynamics for Large Language Models
Nanavati, J., Patel, & Patel, U. (2026) · International Conference on System, Computation, Automation and Networking (ICSCAN) · doi:10.1109/icscan66520.2026.11588326
An Overview of LLM-Based AI Chatbots
Rista, A., Xhaferra, E., & Mukli, L. (2026) · MIPRO ICT and Electronics Convention (MIPRO) · doi:10.1109/mipro70003.2026.11591907
Drivers of Secure and Correct Code: A Factorial Study of Size, Pre-Training, and Data Quality
Liguori, P., Krasniqi, R., & Cotroneo, D. (2026) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w70714.2026.00031
An LLM-based multi-agent framework for automated reliability analysis of spatially variable soils
Guo, S., He, X., Xiong, H., Xu, H., & Sheng, D. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.105065
Huang, Y., Feng, X., Fu, J., Feng, X., Li, B., Ye, Z., Qin, L., Fei, H., Ng, S.-K., & Qin, B. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.773
“While scaling laws suggest that larger models trained with more compute usually achieve higher accuracy and lower error (Kaplan et al., 2020; Hoffmann et al., 2022), we see substantial per-benchmark variability: Qwen3-32B underperforms Qwen3-14B on TriviaQA, and Qwen3-14B underperforms Qwen3-8B on…”
Adaptive Mixing of Embeddings from Multiple Code Language Models for Fault Localization
Yang, J., Park, E. Y., & Ko, I. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00044
“Given that the computational cost of language models increases linearly with their size [52], [53], using several CodeLMs with only a few hundred million parameters can be more cost-efficient than relying on a single LLM such as ChatGPT with 175 billion parameters.”
RADS-PDD: Reproducibility-aware dynamic similarity-based pre-training data detection against LLMs
Fan, X., Miyamoto, R., Mo, F., Chen, C., Matsumoto, T., Kido, F., & Yamana, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109393 · Cited by 1
Liu, L., Yao, S., Liu, Y. P., Wang, Y., Xia, H., & Tang, X. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105027
Mixture of Experts Approach for Domain Adaptation in Finance
Suzuki, M., Sakaji, H., & Izumi, K. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p1093
Sangariyavanich, E., Pakkaranang, B., Sanguanlost, S., et al. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100443
Privacy-Preserving Entity Transformation: A Semantic Compatibility Framework for Language Models
Desai, A., & Prakash, N. (2026) · International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise70044.2026.11607655
“Model training follows compute-optimal scaling principles [9].”
Memory-Efficient Adapter Quantization for On-Device Language Models
Prakash, N., & Desai, A. (2026) · International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise70044.2026.11607491
“…to several factors: (1) the adapter’s small size makes it less sensitive to precision reduction compared to large language models [20], [23], [24], (2) character-level features learned by the adapter are inherently robust to numerical precision variations, and (3) the personalized nature of…”
Jinhao, 高. G., Linxue, 朱. Z., Liangchuan, 李. L., Ji, 周. Z., & Xiangjun, 忻. X. (2026) · Laser & Optoelectronics Progress · doi:10.3788/lop260926
Four Paradigms and Their Limits
Chang, E. Y. (2026) · ACM eBooks · doi:10.1145/3822380.3822383
Introduction: From Pattern Alchemy to Coordination Physics
(2026) · ACM eBooks · doi:10.1145/3822380.3822382
The Impact of the Dataset on Fine-Tuning Multilingual BERT for Multiclass Classification Task
Abreu-Hernández, E., Martínez-Estévez, R., Savón-Berenguer, A., & Moraru, S.-A. (2026) · International Conference on Electronics, Computers and Artificial Intelligence (ECAI) · doi:10.1109/ecai69016.2026.11613801
“DeepMind’s Chinchilla study [13] showed that, under a fixed computing budget, there is an optimal model size associated with a proportional amount of data, revealing that many LLMs had been trained with insufficient data for their scale.”
SeqFedRPC : Sequential Federated Learning With Regularized Parameter Clustering
Mao, P., Du, T. Y., Wu, Z., Wang, D., & Xiao, H. (2026) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.70477
“However, as scaling laws [ 2, 3] drive the trend toward larger models and datasets, FL faces significant challenges regarding the communication overhead associated with aggregating large-scale model parameters across numerous clients [ 4].”
Zhao, H., Mao, S., & Chen, D. (2026) · International Conference on Computer Intelligence and Software Engineering (CICSE) · doi:10.1109/cicse70286.2026.11620946
“Large Language Models (LLMs) have achieved remarkable success across natural language processing tasks [1]–[4], with their capabilities scaling predictably as model parameters and training data expand [5], [6].”
AI Research Automation, Adjustment Frictions, and Growth TransitionsPreprint
Rajhi, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7199111
ModernBERT-TR: A Modern Encoder Foundation Model for TurkishPreprint
Alkurdi, B., Kesgin, H. T., Yuce, M. K., & Amasyalı, M. F. (2026) · Research Square · doi:10.21203/rs.3.rs-9500377/v1 · Cited by 1
Rethinking Data Composition for Generalizable Deepfake Detection
Liu, Y., & Jiang, J. (2026) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase70763.2026.11619428
“Understanding these data-driven patterns is crucial for guiding training decisions [22] and enabling more effective resource allocation during model development [23].”
AI Governance: Data and Ethics
C I C C Research CICC Global Institute (2026) · Springer eBooks · doi:10.1007/978-981-92-3270-3_6
Challenges and Pathways for China’s AI Development
C I C C Research CICC Global Institute (2026) · Springer eBooks · doi:10.1007/978-981-92-3270-3_7
Anxiety over Energy Consumption of AI: Growth Limits and the Green Dilemma
C I C C Research CICC Global Institute (2026) · Springer eBooks · doi:10.1007/978-981-92-3270-3_3
Obfuscation by Simplification: How LLMs Transform Code Through Structural ExpansionPreprint
Tomasi, L. D., Sipio, C. D., Marco, A. D., & Nguyen, P. T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7211169
What Matters for Grocery Product Retrieval with Open Source Vision Language Models
Maminta, E. G., & Atienza, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31666-0_18
AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control
Bui, Q.-H., Pham, B. N., Tran, A.-M., Le, T. T. M., Le, T. P., Nguyen, T. D., & Ta, A. S. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-2590-3_35
Dialectical systems framework for LLM applications in intelligent construction and operation
Xu, X., & Li, Z. (2026) · Automation in Construction · doi:10.1016/j.autcon.2026.107186
The means of prediction and the production function of AI
Kasy, M. (2026) · Journal of Economic Interaction and Coordination · doi:10.1007/s11403-026-00491-5
Vatsal, S., Dubey, H., & Bajaj, A. (2026) · IEEE Access · doi:10.1109/access.2026.3720396
“Scaling studies [1], [2] showed that predictably improving performance can often be achieved by increasing model size, data and compute under empirical scaling laws.”
Closed-Form Residual-Space Rotation for Offline Transformer Width GrowthPreprint
Ramasubramanian, B. (2026) · Research Square · doi:10.21203/rs.3.rs-10580431/v1
Mixture-of-Experts: Efficient Scaling to Trillion-Parameter Models
B, B. P. (2026) · Eduschool Journal of Artificial Intelligence Research (EJAIR) · doi:10.63090/ejair/3139.7336.0004
Mu, S., Jiang, Y.-R., Wu, S., Hong, S., Sha, T., Zhang, J., Zhu, J., Chen, Z., Wang, Z., & Lin, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818445
“Inspired by the success of scaling laws in large language models (LLMs) [8, 10], the scaling laws of recommendation models have recently been widely studied and have achieved significant success in real-world industrial recommendation systems [6, 9, 30, 33, 34, 38, 41].”
The Scaling Paradox in Human-AI CollaborationPreprint
Qi, A., & Wang, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7207559
Harnessing Photonics for Machine Intelligence [Feature]
Zhu, H., Ning, S., Zhou, H., Yin, Z., Chen, R. T., Gu, J., & Pan, D. Z. (2026) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2026.3688606 · Cited by 5
“Underpinned by empirical scaling laws [3, 4], this evolution has shifted the field from isolated algorithmic novelty to systematic scaling of parameters and data, a trajectory that demands vastly more compute.”
AzText: Curating Web-Scale Pretraining Data for a Low-Resource Language
Mahammadli, E., & Rustamov, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31319-5_23
Hassan, M. T., Ahmed, J., & Awais, M. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3839365 · Cited by 3
“As demonstrated by the seminal ”Chinchilla” scaling laws [20], model performance is driven as much by the number of training tokens as by parameter count.”
Metadata
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., & Sifre, L.
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133225605
- Cited by
- 932
- References
- 78
- Citations by year
- 2026: 242 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26
- FWCI
- 96.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.