Training Compute-Optimal Large Language Models
Open accessAbstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent advances in modeling and hardware have increased the data training capacity of LLMs from Wikipedia-scale corpora to close to three orders of magnitude more.
“However, whereas recent advances in modeling and hardware have increased the data training capacity of LLMs, increasing from Wikipedia-scale corpora to close to three orders of magnitude more”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as Hoffmann et al. [71]
The Introduction (p. 2) reports LLMs trained on roughly 300 billion tokens and Chinchilla on 1.4 trillion, supporting the point that LLMs can train on very large datasets. Appendix A (p. 22) lists Wikipedia as one small component of Chinchilla’s dataset, but the paper does not establish a historical increase from Wikipedia-scale corpora to nearly three orders of magnitude more data, or attribute such an increase to advances in modeling and hardware. (LLM, conf=0.82)
Experimental — accuracy not yet measuredOnly well-resourced actors can create cutting-edge foundation models.
“The costs required to create such models (and continuing returns to scale) means that only well-resourced actors can create cuttingedge models (Epoch, 2023; Hoffmann et al., 2022; Kaplan et al., 2020), making them relatively few in number.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Hoffmann et al., 2022
The Introduction (p. 1) says training large language models has substantial compute and energy costs that rise with model size. The optimal-scaling discussion (pp. 7–8) also describes increasing compute requirements. The paper does not establish that only well-resourced actors can create cutting-edge models, or that such actors are few in number. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe largest Llama 3's lower parameter count compared to PALM is due to better understanding of scaling laws.
“Despite containing 405B parameters, our largest Llama 3 in fact contains fewer parameters than earlier and much less performant models such as PALM (Chowdhery et al., 2023), due to better understanding of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).”
From The Llama 3 Herd of Models (2024) · cites this paper as Hoffmann et al., 2022
Section 3.4 (p. 8) concludes that, for many training-compute budgets, smaller models trained on more tokens would perform better. Section 5 (p. 15) reports that this approach yielded a smaller, better-performing Chinchilla relative to Gopher. The paper does not address Llama 3 or PaLM, so it cannot establish that scaling-law insights caused the specific parameter-count difference claimed here. (LLM, conf=0.85)
Experimental — accuracy not yet measuredDeep learning AI and LLMs AI models exhibit similar power scaling laws.
“Given that deep learning AI and LLMs AI models exhibit similar power scaling laws, our focus can be directed specifically towards LLMs AI without a significant loss of generality.”
From Large language models at work in China’s labor market (2025) · cites this paper as Hoffmann et al., 2022
The Introduction (p. 1) describes a power-law relationship between language-model size and performance, and the Discussion (p. 15) discusses power-law scaling for large language models. The paper does not compare those laws with deep-learning models generally, so it does not establish the claimed similarity or justify treating LLMs as representative of all deep learning AI. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPrior work assumes that the necessary data to support scaling is unlimited.
“Prior work (Kaplan et al., 2020;Hoffmann et al., 2022) assumes that the necessary data to support scaling is unlimited.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The Introduction (p. 2) says the analysis treats its training runs as being in an “infinite data regime” because they use fewer tokens than the corpus contains. The paper also projects increasing data needs as compute scales (Optimal model scaling, p. 8), but it does not assume that the data needed for scaling is literally unlimited; it highlights the need for dataset collection. (LLM, conf=0.82)
Experimental — accuracy not yet measured
For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from Hoffmann et al. has U_N of approximately 7M parameters.
“For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from (Hoffmann et al., 2022) has U N of approximately 7M parameters (see §C for the scaling coefficients we use).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Section 3.4 (p. 8) supports using Hoffmann et al.’s scaling laws to relate optimal model size to training tokens. It does not give a 100M-token, one-epoch estimate: its Table 3 pairs 400M parameters with 8.0B tokens, which extrapolates to roughly 5M—not 7M—parameters at 100M tokens. That extrapolation is also below the model sizes used to fit the laws (Section 3, p. 4). (LLM, conf=0.82)
Experimental — accuracy not yet measuredThis is very close to the 73 billion parameters and 1.3 trillion tokens predicted by the IsoFLOP curves on C4 from Hoffmann et al. (2022).
“This is very close to the 73 billion parameters and 1.3 trillion tokens predicted by the IsoFLOP curves on C4 from (Hoffmann et al., 2022)”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
The Appendix (p. 22) says the authors generated C4 IsoFLOP curves showing parameter and token scaling. Our copy gives scaling exponents but does not state the claimed prediction of 73 billion parameters and 1.3 trillion tokens; Figure A2 appears only as an image placeholder. (LLM, conf=0.85)
Experimental — accuracy not yet measuredBased on their IsoFLOP training runs on C4, Hoffmann et al. only provide the information that for C4, compute allocated to data and parameters should be scaled exactly equally for optimality, i.e. a = b = 0.5.
“Based on their IsoFLOP training runs on C4, they only provide the information that for C4, compute (C) allocated to data (D) and parameters (N ) should be scaled exactly equally for optimality, i.e. a = b = 0.5”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Appendix Table A2 (p. 22) reports C4 IsoFLOP scaling coefficients of a = 0.50 and b = 0.50. It labels them estimates, however, so the paper does not establish that exactly equal scaling is required for optimality. The appendix also provides C4 IsoFLOP curves, rather than only the coefficients. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe precise ratios of model parameters and training tokens vary considerably across data sets and approaches.
“the precise ratios vary considerably across data sets and approaches.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Appendix D.3 (p. 26) gives different optimal token counts for the same model sizes under Approaches 2 and 3, supporting variation across approaches. But Appendix C (p. 22) says scaling on C4 and GitHub is very similar to MassiveText and suggests the results are dataset-independent under its stated condition. The claim overstates variation across datasets. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGiven the Gopher compute budget of 5.76 × 10^23 FLOPs, Hoffmann et al.'s parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.
“For example given the Gopher (Rae et al., 2021) compute budget of 5.76 × 10 23 FLOPs, their parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Section 3.3 (p. 7) says the fitted parametric loss function projects 40 billion parameters as optimal for Gopher’s compute budget; Section 3 (p. 4) gives that budget as 5.76 × 10^23 FLOPs. But Appendix A (p. 22) says the scaling runs used MassiveText, a mixture containing MassiveWeb, not MassiveWeb alone. (LLM, conf=0.90)
Experimental — accuracy not yet measuredIf the training data set is C4, Hoffmann et al.'s IsoFLOP approach predicts 73 billion parameters to be optimal.
“Meanwhile, if the training data set is C4 (Raffel et al., 2020) their IsoFLOP approach predicts 73 billion parameters to be optimal, almost twice as much.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Section 3.4 (p. 8) says the authors reproduced their IsoFLOP analysis on C4, and the Appendix (p. 22) reports C4 scaling results. Neither passage gives a C4-specific optimum of 73 billion parameters; the numerical prediction attributed to the paper is unsupported by our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe creators of Galactica trained a 120 billion parameter model on 450 billion tokens, a significant overallocation to parameters even in Chinchilla terms.
“The creators of Galactica decided to train a 120 billion parameter model on 450 billion tokens, a significant overallocation to parameters even in Chinchilla terms (black efficient frontier).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Section 3.4 (p. 8) gives a compute-optimal allocation of roughly 20 training tokens per parameter and says large models trained on fewer tokens are oversized. That supports the comparison to the Chinchilla frontier, but our copy does not identify Galactica or establish that its creators trained a 120-billion-parameter model on 450 billion tokens (searched: Galactica, 120 billion, 450 billion). (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe default Chinchilla scaling law predicts loss to continue decreasing as parameters are added, which is in stark contrast to the empirical data.
“Meanwhile, the default Chinchilla scaling law (Hoffmann et al., 2022) predicts loss to continue decreasing as parameters are added, which is in stark contrast to the empirical data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Section 3.3 (pp. 6–7) models a loss term that decreases as parameter count increases, but that prediction holds when training tokens are held fixed. Under a fixed compute budget, the paper instead describes an optimal balance between parameters and tokens; it does not report the claimed stark contrast with empirical data. (LLM, conf=0.83)
Experimental — accuracy not yet measuredChinchilla 70B outperformed other large language models on a large collection of language tasks.
“We conducted our study using Chinchilla 70B because it outperformed other LLMs on a large collection of language tasks [50].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [50]
Our findings suggest that evolution of embodied intelligent behavior, biological or artificial, may require coordinated scaling of visual acuity and processing.
“While AI systems focus on scaling neural, data, and computation capacity (30, 31, 74–77), our findings suggest that evolution of embodied intelligent behavior (biological or artificial) may require coordinated scaling of visual acuity and processing.”
From What if eye...? Computationally recreating vision evolution (2025) · cites this paper as [74]
Recent investigations on large language models have profiled the trade-off between model size and data volume when the compute budget is fixed, suggesting model size is another important dimension for pre-training worthy of further investigation for human sensing data.
“Recent investigations on large language models have profiled the trade-off between model size and data volume when the compute budget is fixed42, suggesting model size is another important dimension for pre-training worthy of further investigation for human sensing data.”
From Self-supervised learning for human activity recognition using 700,000 person-days of wearable data (2024) · cites this paper as [42]
The capacity and capability of LLMs greatly increased with scale, taking the same path that nature took by evolving bigger and better brains.
“The capacity and capability of LLMs greatly increased with scale, taking the same path that nature took by evolving bigger and better brains (Allman, 1999; Hoffmann et al., 2022).”
From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Hoffmann et al., 2022
The quantity of data is a strong factor in a trained model's performance on evaluation tasks.
“recent work has put a strong emphasis on the quantity of data being a strong factor in a trained model's performance on evaluation tasks (Kaplan et al., 2020; Hoffmann et al., 2022)”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Hoffmann et al., 2022
Scaling laws revealed predictable relationships between parameter count, data volume, and loss.
“Scaling laws revealed predictable relationships between parameter count, data volume, and loss, reinforcing the intuition that progress tracked directly with model size [Kaplan et al., 2020, Hoffmann et al., 2022].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Hoffmann et al., 2022
Scaling laws numerically uncover a power-law relationship between model size, data volume, training scale and the growth of model performance.
“The impact of scaling is illustrated by the scaling laws [34, 35], which numerically uncover the power-law relationship between model size, data volume, training scale and the growth of the model's performance.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [35]
With increased model size, given adequate data and computational resources, proficiency in word prediction within textual contexts is enhanced.
“With increased model size, given adequate data and computational resources, proficiency in word prediction within textual contexts, as demonstrated by the training dataset, is enhanced (Kaplan et al. 2020; Ganguli et al. 2022; Hoffmann et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Hoffmann et al. 2022
Hoffmann et al. [24] models the improvement for larger model training budgets as a power function, proportional to the model parameters N and training tokens D.
“Hoffmann et al. [24] models the improvement for larger model training budgets as a power function, proportional to the model parameters N and training tokens D , with the exact functional form and prediction setup varying between work [44].”
From Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (2025) · cites this paper as [24]
The 70-billion parameter Chinchilla model outperforms the 280-billion parameter Gopher model while using a similar compute budget by being trained on four times more data.
“Notably, the 70-billion parameter Chinchilla model (Hoffmann et al., 2022) outperforms the 280-billion parameter Gopher model (Rae et al., 2021) while using a similar compute budget by being trained on four times more data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Allocation is balanced: resources are divided roughly equally between scaling of parameters and data.
“Allocation is balanced: resources are divided roughly equally between scaling of parameters and data (Hoffmann et al., 2022).”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Chinchilla scaling laws conclude that α ≈ β and hence N and D should be scaled proportionally for compute-optimal training.
“These methods lead to the conclusion that α ≈ β and hence N and D should be scaled proportionally for compute-optimal training.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Hoffmann et al. (2022) have shown that the equal scaling of model parameters and training tokens holds across different training data sets.
“While Hoffmann et al. (2022) have shown that the equal scaling of model parameters and training tokens holds across different training data sets”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
For C4, Hoffmann et al. do not provide the coefficients necessary to compute loss with their parametric loss function.
“However, for C4, which is our training data set, they do not provide the coefficients necessary to compute loss with their parametric loss function.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al. (2022)
Results from Hoffmann et al. (2022) show that there are differences in the scaling coefficients when training on C4 as compared to GitHub code, however, the overall allocation trend is the same.
“Results from (Hoffmann et al., 2022) show that there are differences in the scaling coefficients when training on C4 as compared to GitHub code, however, the overall allocation trend is the same.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
The parameter counts reported in Chinchilla (Hoffmann et al., 2022) are significantly different from the authors' calculations, especially at larger scales.
“We find the parameter counts reported in Chinchilla (Hoffmann et al., 2022) to be significantly different than our calculations, especially at larger scales.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Hoffmann et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
This paper’s reference list could not be split into single entries.
Cited by
Sorted by citations
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“, 2021), Chinchilla (Hoffmann et al., 2022), Megatron–Turing NLG (Smith et al.”
BackgroundMethodResultInfluentialDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
BackgroundAutonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“Recent advancements in LM research [30, 38, 75, 170, 205] are rooted in the capacity to increase LM size in terms of number of”
BackgroundSelf-citationNucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 656 · see citing works
“We extended the training duration of the 250M-and 500M-parameter models to encompass 1 trillion tokens, aligning with recent recommendations in the literature 45 (Fig.”
Reproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523 · see citing works
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
“The study of scaling laws [3, 9, 10, 11, 12, 13] relate these compute trends to model performance, and are actively used by large corporations to inform their training resource requirements [2, 14].”
BackgroundMethodA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.507 · Cited by 316 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“Large Language Models (LLMs) In recent years, tremendous resources have been allocated to scale Transformer-based language models (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022; Hoffmann et al., 2022; Smith et al., 2022; Zhang et al., 2022; Lieber et al., 2021; Fedus et al., 2021…”
BackgroundGenerative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · see citing works
MM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260 · see citing works
Jablonka, K. M., Ai, Q., Al-Feghali, A., Badhwar, S., Bocarsly, J. D., Bran, A. M., Bringuier, S., Brinson, L. C., Choudhary, K., Circi, D., et al. (2023) · Digital Discovery · doi:10.1039/d3dd00113j · Cited by 218 · see citing works
AI generates covertly racist decisions about people based on their dialect
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · Nature · doi:10.1038/s41586-024-07856-5 · Cited by 233 · see citing works
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202 · see citing works
What large language models know and what people think they know
Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., & Smyth, P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00976-7 · Cited by 209 · see citing works
Larger and more instructable language models become less reliable
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024) · Nature · doi:10.1038/s41586-024-07930-y · Cited by 203 · see citing works
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 195 · see citing works
“The 1.3-billion parameter models were trained for 30 billion tokens, while the 4-billion parameter LLMs received 100 billion tokens; both setups were consistent with compute-optimal scaling laws 29 .”
InfluentialAI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · see citing works
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169 · see citing works
Large language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161 · see citing works
Choudhury, A., & Chaudhry, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/56764 · Cited by 159 · see citing works
Self-supervised learning for human activity recognition using 700,000 person-days of wearable data
Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01062-3 · Cited by 166 · see citing works
“The test performance for the AoT stayed at the random chance level, and the test performance for permutation dropped roughly 10 percentage points without weighted sampling.”
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.09419 · Cited by 574 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142 · see citing works
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
Language Models and Cognitive Automation for Economic Research
Korinek, A. (2023) · doi:10.3386/w30957 · Cited by 129 · see citing works
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109 · see citing works
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299 · see citing works
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94 · see citing works
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Novel machine learning approaches revolutionize protein knowledge
Bordin, N., Dallago, C., Heinzinger, M., Kim, S., Littmann, M., Rauer, C., Steinegger, M., Rost, B., & Orengo, C. (2023) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2022.11.001 · Cited by 72 · see citing works
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76 · see citing works
Distilling large language models for matching patients to clinical trials
Nievas, M., Basu, A., Wang, Y., & Singh, H. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae073 · Cited by 67 · see citing works
Structured information extraction from complex scientific text with fine-tuned large language models
Dunn, A., Dagdelen, J., Walker, N., Lee, S., Rosen, A. S., Ceder, G., Persson, K., & Jain, A. (2022) · arXiv · doi:10.48550/arxiv.2212.05238 · Cited by 67 · see citing works
DeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113 · see citing works
Lyu, W., Zhang, S., Chung, T., Sun, Y., & Zhang, Y. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100383 · Cited by 68 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
“4 trillion tokens (Hoffmann et al., 2022) and dataset creation pipelines representing a significant part of large language model projects.”
BackgroundKuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47 · see citing works
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46 · see citing works
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
“[71] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Ja…”
BackgroundEnhanced automated code vulnerability repair using large language models
de-Fitero-Dominguez, D., Garcia-Lopez, E., Garcia-Cabot, A., & Martinez-Herraiz, J.-J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109291 · Cited by 43 · see citing works
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49 · see citing works
Studying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40 · see citing works
MELTing Point: Mobile Evaluation of Language Transformers
Laskaridis, S., Katevas, K., Minto, L., & Haddadi, H. (2024) · Proceedings of the 30th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3636534.3690668 · Cited by 37 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49 · see citing works
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., Winter, C., Arnold, M., Ó hÉigeartaigh, S., Korinek, A., et al. (2023) · doi:10.2139/ssrn.4596436 · Cited by 31 · see citing works
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28 · see citing works
Self-citationAlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs.
Ghosh, M., Mukherjee, S., Ganguly, A., Basuchowdhuri, P., Naskar, S. K., & Ganguly, D. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.005 · Cited by 27 · see citing works
The rising costs of training frontier AI models
Cottier, B., Rahman, R., Fattorini, L., Maslej, N., Besiroglu, T., & Owen, D. (2024) · arXiv · doi:10.48550/arxiv.2405.21015 · Cited by 27 · see citing works
Computing Power and the Governance of Artificial Intelligence
Sastry, G., Heim, L., Belfield, H., Anderljung, M., Brundage, M., Hazell, J., O'Keefe, C., Hadfield, G. K., Ngo, R., Pilz, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.08797 · Cited by 26 · see citing works
Inverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25 · see citing works
ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis
Liu, Y., Peng, X., Du, T., Yin, J., Liu, W., & Zhang, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.476 · Cited by 22 · see citing works
Large Language Models Can Be Contextual Privacy Protection Learners
Xiao, Y., Jin, Y., Bai, Y., Wu, Y., Yang, X., Luo, X., Yu, W., Zhao, X., Liu, Y., Gu, Q., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.785 · Cited by 23 · see citing works
Large language models at work in China’s labor market
Chen, Q., Ge, J., Xie, H., Xu, X., & Yang, Y. (2025) · China Economic Review · doi:10.1016/j.chieco.2025.102413 · Cited by 17 · see citing works
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14 · see citing works
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2022) · arXiv · doi:10.48550/arxiv.2209.14958 · Cited by 10 · see citing works
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
Scaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9 · see citing works
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Chen, X., Sun, Z., Wenjin, G., Zhang, M., Chen, Y., Sun, Y., Su, H., Pan, Y., Klakow, D., Li, W., et al. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.782 · Cited by 7 · see citing works
Artsi, Y., Klang, E., Collins, J. D., Glicksberg, B. S., Korfiatis, P., Nadkarni, G. N., & Sorin, V. (2025) · medRxiv · doi:10.1101/2025.03.18.25324193 · Cited by 7 · see citing works
What if eye...? Computationally recreating vision evolution
Tiwary, K., Young, A., Tasneem, Z., Klinghoffer, T., Dave, A., Poggio, T., Nilsson, D.-E., Cheung, B., & Raskar, R. (2025) · Science Advances · doi:10.1126/sciadv.ady2888 · Cited by 4 · see citing works
AgCV: An Agentic framework for automating computer vision application
Saxena, A., Chaudhari, A. Y., & Gupta, A. (2025) · MethodsX · doi:10.1016/j.mex.2025.103424 · Cited by 4 · see citing works
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3 · see citing works
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1 · see citing works
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 1 · see citing works
Alviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
Towards Embodiment Scaling Laws in Robot Locomotion
Ai, B., Dai, L., Bohlinger, N., Li, D., Mu, T., Wu, Z., Fay, K., Christensen, H. I., Peters, J., & Su, H. (2025) · arXiv · doi:10.48550/arxiv.2505.05753
Gao, X., Xie, W., Lixin, Fang, X., Hang, C., Li, C., Lin, Y., & Xu, X. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i25.39266
The AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
Scaling Data-Constrained Language Models with Synthetic Data
Kiyomaru, H., Oda, Y., Kodama, T., Liu, C., & Kawahara, D. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.52
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data
Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2022) · arXiv · doi:10.48550/arxiv.2206.02909
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13 · see citing works
- Authors
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. V. D., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., & Sifre, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 30016-30030
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2203.15556
- OpenAlex
- W7133225605
- Cited by
- 929
- References
- 0
- Citations by year
- 2026: 239 · 2025: 188 · 2024: 248 · 2023: 226 · 2022: 26 · 2019: 2
- FWCI
- 96.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- large language models, computational efficiency, training efficiency, model compression, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
