The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Open accessAbstract
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs are particularly vulnerable to performance degradation from low-quality data containing spam, noise, and duplicates [61,62].”
From A Survey of Large Language Models (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla stored a higher-precision copy of the weights in the sharded optimiser state.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example given the Gopher (Rae et al., 2021) compute budget of 5.76 × 10 23 FLOPs, their parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Small Gopher models also do not perform above random until scaled up to the largest model of 5 · 10 23 training FLOPs (280B parameters), for which performance jumps to more than 20% above random (Rae et al., 2021).”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For GPT-3, Gopher, and Chinchilla, models of ∼ 10 22 training FLOPs ( ∼ 10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to 3-5 · 10 23 training FLOPs (70B-280B parameters) enables performance to substantially surpass random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture (Hoffmann et al. 2022; Rae et al. 2021).”
From Algorithmic progress in language models (2024) · cites this paper as Rae et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021). In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“in the analysis of mod- els trained on the same corpus in Rae et al. (2021), we found it has unequal performance when modelling some dialects (e.g., African American English).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“The same dataset is used as in Rae et al. (2021).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Notably, the 70-billion parameter Chinchilla model (Hoffmann et al., 2022) outperforms the 280-billion parameter Gopher model (Rae et al., 2021) while using a similar compute budget by being trained on four times more data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The TruthfulQA benchmark (Lin et al., 2021) showed that GPT-3 models were more likely to mimic human falsehoods as they got larger, though Rae et al. (2021) later showed on a multiple-choice version that scaling Gopher to 280B enabled emergent performance substantially better than random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“| GitHub | 0.53 | 0.47 |”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 1
- [10]10. O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-SNLI: Natural language inference with natural language explanations. arXiv preprint arXiv:1812.01193, 2018. - Y. T. Cao and H. Daumé. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecyle. Computational Linguistics, pages 1-47, 2021. - N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019. - N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models, 2021. - N. Chater. The search for simplicity: A fundamental cognitive principle? The Quarterly Journal of Experimental Psychology Section A, 52(2):273-302, 1999. - H. Chen, X. Liu, D. Yin, and J. Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-35, 2017. - T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015. - P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. - D. Coyle and A. Weller. 'Explaining' machine learning reveals policy challenges. Science, 368(6498): 1433-1434, 2020. · arXiv
Cited by
Sorted by citations
Items 201–300 of 336
ChatGPT Alternative Solutions: Large Language Models Survey
Alipour, H., Pendar, N., & Roy, K. (2024) · doi:10.5121/csit.2024.1405114 · Cited by 13
Walter, Y. (2023) · AI and Ethics · doi:10.1007/s43681-023-00276-7 · Cited by 14
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y. W., & Gui, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.35 · Cited by 14
On the Role of Bidirectionality in Language Model Pre-Training
Artetxe, M., Du, J., Goyal, N., Zettlemoyer, L., & Stoyanov, V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.293 · Cited by 19
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2022) · arXiv · doi:10.48550/arxiv.2209.14958 · Cited by 10
The role of AI for MRI-analysis in multiple sclerosis—A brief overview
Falet, J.-P. R., Nobile, S., Szpindel, A., Barile, B., Kumar, A., Durso-Finley, J., Arbel, T., & Arnold, D. L. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1478068 · Cited by 14
Reasoning in Large Language Models Through Symbolic Math Word Problems
Gaur, V., & Saunshi, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.364 · Cited by 25
Inferring Implicit Relations in Complex Questions with Language Models
Katz, U., Geva, M., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.188 · Cited by 13
Responsible scaling of artificial intelligence in healthcare: standardization meets customization
Lukkien, D., Nap, H. H., Peine, A., Minkman, M. M. N., Moors, E. H. M., & Boon, W. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09842-5 · Cited by 9
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing
Alghamdi, A., Duan, X., Jiang, W., Wang, Z., Wu, Y., Xia, Q., Wang, Z., Zheng, Y., Rezagholizadeh, M., Huai, B., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.181 · Cited by 9
Lingualyzer: A computational linguistic tool for multilingual and multidimensional text analysis
Linders, G. M., & Louwerse, M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02284-1 · Cited by 8
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models
Michaelov, J. A., Arnett, C., Chang, T. A., & Bergen, B. K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.227 · Cited by 18
Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards
Chai, Y., Wang, S., Sun, Y. M., Tian, H., Wu, H., & Wang, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.8 · Cited by 14
Context Generation Improves Open Domain Question Answering
Su, D., Patwary, M. A., Prabhumoye, S., Xu, P., Prenger, R., Shoeybi, M., Fung, P., Anandkumar, A., & Catanzaro, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.60 · Cited by 12
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8
Benchmarking Large Language Model Capabilities for Conditional Generation
Maynez, J., Agrawal, P., & Gehrmann, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.511 · Cited by 21
BMCook: A Task-agnostic Compression Toolkit for Big Models
Zhang, Z., Gong, B., Chen, Y., Han, X., Zeng, G., Zhao, W., Chen, Y., Liu, Z., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.40 · Cited by 8
Lee, H.-R., & Kim, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4394715/v1 · Cited by 8
Ho-tin, W., & Yip, G.-L. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/wkb32 · Cited by 8
Pujiono, I., Agtyaputra, I. M., & Ruldeviyani, Y. (2024) · JITK (Jurnal Ilmu Pengetahuan dan Teknologi Komputer) · doi:10.33480/jitk.v10i1.5572 · Cited by 22
Derivational morphology reveals analogical generalization in large language models
Hofmann, V., Weissweiler, L., Mortensen, D. R., Schütze, H., & Pierrehumbert, J. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2423232122 · Cited by 21
Ethical and Epistemological Reflections on Autonomous AI-powered Agents (AAIAs)
Farina, M., Wang, Y., & Kladko, S. (2025) · Topoi · doi:10.1007/s11245-025-10218-z · Cited by 9
Wibæk, R., Andersen, G. S., Dahm, C. C., Witte, D. R., & Hulmán, Á. (2023) · JMIR Medical Informatics · doi:10.2196/43638 · Cited by 7
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Csordás, R., Irie, K., & Schmidhuber, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.49 · Cited by 19
EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
Tang, H., Sun, Y., Wu, D., Liu, K., Zhu, J., & Kang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.565 · Cited by 17
Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks
Wang, Z., Pan, X., Yu, D., Yu, D., Chen, J., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.246 · Cited by 13
Probabilistic Inference Layer Integration in Mistral LLM for Accurate Information RetrievalPreprint
Wang, B., Wang, S., & Ouyang, Q. (2024) · Research Square · doi:10.21203/rs.3.rs-3828707/v1 · Cited by 7
From errors to excellence: AI-driven scaffolding for advancing L2 writing skills
Ren, B., Dong, Y., & Zhang, B. (2025) · Cogent Education · doi:10.1080/2331186x.2025.2588512 · Cited by 12
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models
Pozzobon, L., Ermiş, B., Lewis, P., & Hooker, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.339 · Cited by 19
Evaluating Quantized Llama 2 Models for IoT Privacy Policy Language Generation
Malisetty, B., & Pérez, A. J. (2024) · Future Internet · doi:10.3390/fi16070224 · Cited by 9
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Zhang, Y., Yasunaga, M., Zhou, Z., HaoChen, J. Z., Zou, J., Liang, P., & Yeung, S. Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.472 · Cited by 17
A Universal Discriminator for Zero-Shot Generalization
Xu, H., Lin, Z., Zhou, J., Zheng, Y., & Yang, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.589 · Cited by 12
Better Quality Pre-training Data and T5 Models for African Languages
Oladipo, A., Adeyemi, M., Ahia, O., Owodunni, A., Ogundepo, O., Adelani, D. I., & Lin, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.11 · Cited by 9
Reinforcement Learning for Optimized Information Retrieval in LLaMAPreprint
Tu, C.-H., Hsu, H.-J., & Chen, S.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-3847100/v1 · Cited by 7
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2209.01188 · Cited by 6
Predictive design of crystallographic chiral separation
Elijošius, R., King‐Smith, E., Faber, F. A., Bernier, L., Berritt, S., Farrell, W. P., Hou, X., Klug-McLeod, J. L., Mustakis, J. G., Sach, N. W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62825-4 · Cited by 8
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
Deng, H., & Raffel, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.721 · Cited by 5
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 13
kNN-LM Does Not Improve Open-ended Text Generation
Wang, S., Song, Y., Drozdov, A., Garimella, A., Manjunatha, V., & Iyyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.929 · Cited by 12
Knowledge-based Word Tokenization System for Urdu
Khan, W., Khan, A., Khan, K., Khan, S. N., & Haq, R. (2024) · Journal of Informatics and Web Engineering · doi:10.33093/jiwe.2024.3.2.6 · Cited by 5
Ethical Considerations and Best Practices for Using Large Language Models in Decision - Making
Saxena, A. K. (2024) · International Journal of Science and Research (IJSR) · doi:10.21275/sr24710062038 · Cited by 6
Quati: A Brazilian Portuguese Information Retrieval Dataset from Native Speakers
Bueno, M. C., de Oliveira, E. S., Nogueira, R. F., Lotufo, R., & Pereira, J. (2024) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2024.245426 · Cited by 19
Csordás, R., Irie, K., & Schmidhuber, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.662 · Cited by 11
Fundamental Approaches to Software Engineering
Beyer, D., & Cavalcanti, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57259-3 · Cited by 7
Empower LlaMa 2 for Advanced Logical Reasoning in Natural Language UnderstandingPreprint
Watanabe, N., Kinasaka, K., & Nakamura, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4333417/v1 · Cited by 5
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. L., Reinecke, K., & Sap, M. (2023) · arXiv · doi:10.48550/arxiv.2306.01943 · Cited by 31
“In order to evaluate GPT-4, we combine Delphi’s GPT-3 zero-shot evaluation setup from Jiang et al. (2021) with the template used by Rae et al. (2021): Text:”
MethodData-Efficient French Language Modeling with CamemBERTa
Antoun, W., Sagot, B., & Seddah, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.320 · Cited by 6
Not Enough Data to Pre-train Your Language Model? MT to the Rescue!
Urbizu, G., Vicente, I. S., Saralegi, X., & Corral, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.235 · Cited by 4
Kim, G., Lee, H., Kim, D., Jung, H., Park, S., Kim, Y., Yun, S., Kil, T., Lee, B., & Park, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.735 · Cited by 12
Ye, S., Jang, J., Kim, D., Jo, Y., & Seo, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.822 · Cited by 10
Exploring Lottery Prompts for Pre-trained Language Models
Chen, Y., Ding, N., Wang, X., Hu, S., Zheng, H.-T., Liu, Z., & Xie, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.860 · Cited by 8
An Investigation of LLMs’ Inefficacy in Understanding Converse Relations
Qi, C., Li, B., Hui, B., Wang, B., Li, J., Wu, J., & Laili, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.429 · Cited by 7
Artificial General Intelligence (AGI) for Medical Education and TrainingPreprint
Lema, K. G. (2023) · AfricArXiv (OSF Preprints) · doi:10.31730/osf.io/xytzm · Cited by 5
Lu, T., Hu, J., & Chen, P. (2024) · doi:10.36227/techrxiv.171742386.68305769/v1 · Cited by 4
Large language models management of complex medication regimens: a case-based evaluationPreprint
Xu, S. X., Most, A. K., Chase, A. M., Hedrick, T., Murray, B., Keats, K. R., Smith, S., Barreto, E. F., Liu, T., & Sikora, A. N. (2024) · medRxiv · doi:10.1101/2024.07.03.24309889 · Cited by 4
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“…seemed initially promising, systematic studies began to generate negative results across many of these domains [32, 28, 1, 30, 10], claiming that LLM scaling shows much lower returns for reasoning tasks [24], and showcasing brittle performance in the face of minor problem permutations [22, 9, 3].”
BackgroundReliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3
“The recent emergence of large language models (LLMs) [4, 26, 25, 9, 8, 30, 35, 36] has revolutionized the field of Natural Language Processing (NLP), with LLMs demonstrating human-level performance across various professional and academic benchmarks [27] and exhibiting an excellent understanding of…”
Evaluation of Document Deduplication Algorithms for Large Text Corpora
Leveling, J., Helmer, L., Stein, B. J., Wegener, D., Sheikh, Z., Fernandes, E., & Abdelwahab, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82481-4_27 · Cited by 4
PLEASE: Generating Personalized Explanations in Human-Aware Planning
Vasileiou, S. L., & Yeoh, W. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230543 · Cited by 3
Gradient Ascent Post-training Enhances Language Model Generalization
Yoon, D., Jang, J., Kim, S., & Seo, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.74 · Cited by 3
Geographical Erasure in Language Generation
Schwöbel, P., Golebiowski, J., Donini, M., Archambeau, C., & Pruthi, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.823 · Cited by 13
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue
Su, H., Kumar, S. H., Mazumder, S., Chen, W., Manuvinakurike, R., Okur, E., Sahay, S., Nachman, L., Chen, S.-T., & Lee, H.-Y. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.4 · Cited by 3
On the Generalization Ability of Retrieval-Enhanced Transformers
Norlund, T., Doostmohammadi, E., Johansson, R., & Kuhlmann, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.109 · Cited by 8
Most, A. K., Chase, A. M., & Sikora, A. N. (2024) · medRxiv · doi:10.1101/2024.06.29.24309701 · Cited by 3
Malaborbor, R. A. C., Rivera, V. S., Diloy, M. A., Luna, L. R. D., Cruz, A. L. Y. D., & Velasco, A. T. (2024) · International Conference on Digital Technology in Education (ICDTE) · doi:10.1145/3696230.3696247 · Cited by 2
Corpus Complexity Matters in Pretraining Language Models
Agrawal, A., & Singh, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.20 · Cited by 3
Prompterator: Iterate Efficiently towards More Effective Prompts
Sučík, S., Skala, D., Švec, A., Hraška, P., & Šuppa, M. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.43 · Cited by 3
Inteligencia artificial transformación, retos y prospectiva social
de Ensenada, C. D. I. C. Y. D. E. S., García‐Macías, J. A., Rojo, I. D. J. P., California, U. A. D. B., Rivas, D., & California, U. A. D. B. (2024) · doi:10.61728/ae20240998 · Cited by 3
Scaling Laws for BERT in Low-Resource Settings
Urbizu, G., Vicente, I. S., Saralegi, X., Agerri, R., & Soroa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.492 · Cited by 4
The current state of summarization
Retkowski, F. (2023) · AI critique · doi:10.1515/9783839467664-016 · Cited by 6
Estimating Large Language Model Capabilities without Labeled Test Data
Fu, H. Y., Ye, Q., Xu, A., Ren, X., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.639 · Cited by 6
Wodecki, A. (2023) · e-mentor · doi:10.15219/em100.1617 · Cited by 3
DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation
Gu, Z., Sun, X., Lian, F., Kang, Z., Xu, C., & Fan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29768 · Cited by 3
A Survey on Auto-Parallelism of Neural Networks TrainingPreprint
Liang, P., Tang, Y. J., Zhang, X., Bai, Y., Su, T., Qiao, L., Lai, Z., & Li, D. (2022) · doi:10.36227/techrxiv.19522414 · Cited by 2
Rong, B., Gifford, A. T., Düzel, E., & Cichy, R. M. (2025) · eLife · doi:10.7554/elife.108915 · Cited by 3
A Catalog of Transformer Models
D’Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r609337 · Cited by 3
Förster, S. (2023) · A Peer-Reviewed Journal About · doi:10.7146/aprja.v12i1.140444 · Cited by 2
Performance of an Artificial Intelligence Foundation Model for Prostate Radiotherapy Segmentation
Doucette, M., Liao, C.-Y., Lin, M., Jiang, S. B., Nguyen, D. T. D., & Yang, D. X. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00150 · Cited by 2
Moore, L., van de Laar, M., Wong, P., & O’Donoghue, C. (2025) · Land Use Policy · doi:10.1016/j.landusepol.2025.107716 · Cited by 4
Auditing unauthorized training data from AI generated content using information isotopes
Qi, T., Yin, J., Cai, D., Xie, Y., Wang, H., Hu, Z., Yang, P., Nan, G., Zhou, Z., Wu, C., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-68862-x · Cited by 3
Large language models as human proxies
Karetnikov, N., Rahwan, I., & Svetinović, D. (2026) · Nature Computational Science · doi:10.1038/s43588-026-01060-3 · Cited by 1
CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
Dong, Y., Martin, L. J., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.832 · Cited by 2
Spokoyny, D., Wu, C.-S., & Xiong, C. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.5 · Cited by 1
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 5
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy
Wiegreffe, S., Finlayson, M., Tafjord, O., Clark, P. E., & Sabharwal, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.522 · Cited by 4
Prompting Steganography: A New Paradigm
Wu, H. (2024) · Electronic Imaging · doi:10.2352/ei.2024.36.4.mwsf-338 · Cited by 2
Kosenko, D., Kuratov, Y. M., & Zharikova, D. (2023) · Doklady Mathematics · doi:10.1134/s1064562423701168 · Cited by 1
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research
Nityasya, M. N., Wibowo, H. A., Aji, A. F., Winata, G. I., Prasojo, R. E., Blunsom, P., & Kuncoro, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.477 · Cited by 3
Self-citationGAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration
Piktus, A., Ogundepo, O., Akiki, C., Oladipo, A., Zhang, X., Schoelkopf, H., Biderman, S., Potthast, M., & Lin, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.57 · Cited by 4
A Survey on Auto-Parallelism of Neural Networks TrainingPreprint
Liang, P., Tang, Y. J., Zhang, X., Bai, Y., Su, T., Qiao, L., Lai, Z., & Li, D. (2022) · doi:10.36227/techrxiv.19522414.v1 · Cited by 1
Evaluating Accuracy and Reproducibility of Large Language Model Performance in Pharmacy EducationPreprint
Most, A. K., Hu, M., Yang, H., Liu, T., Chen, X., Li, S., Xu, S. X., Liu, Z., & Sikora, A. N. (2024) · medRxiv · doi:10.1101/2024.03.21.24304667 · Cited by 1
A Catalog of Transformer Models
D’Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r606311 · Cited by 1
Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models
Jiang, J., Truong, A. K. Y., Chen, Y., Bao, Q., Wang, S., Chen, P., Wang, J., Kong, L., Li, Y., & Wu, C. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.102 · Cited by 1
Self-citationVisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1
Self-citationSEDM: Selective Self-Evolving Distributed Memory for Agents
Liu, Y., Min, A., Wang, Y., Yao, Y., Han, Y., Zhang, R., He, J., Li, Z., Qi, Y., Xu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.09498 · Cited by 20
“Vector retrieval [17, 20, 15, 23, 14] leverages semantic similarity to identify relevant entries, while hierarchical organization arranges information in layered structures according to abstraction levels [37].”
Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
Metadata
- Authors
- Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. V. D., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., d'Autume, C. D. M., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., Casas, D. D. L., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., & Irving, G.
- ORCID
- Hendricks, Lisa Anne
- Rauh, Maribeth
- Huang, Po-Sen
- Dathathri, Sumanth
- Huang, Saffron
- Higgins, Irina
- Creswell, Antonia
- Budden, David
- Simonyan, Karen
- Paganini, Michela
- Lazaridou, Angeliki
- Mensch, Arthur
- Li, Yujia
- Mikulik, Vladimir
- Babuschkin, Igor
- Johnson, Matthew
- Weidinger, Laura
- Gabriel, Iason
- Vinyals, Oriol
- Ayoub, Kareem
- Hassabis, Demis
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.11446
- OpenAlex
- W4226146865
- Cited by
- 235
- References
- 0
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 52 · 2023: 121 · 2022: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- gophers, large language models, model scaling, transformer models, reading comprehension, fact-checking, toxic language detection, bias in language models, AI safety, mathematical reasoning, training data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
