The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Open accessAbstract
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs are particularly vulnerable to performance degradation from low-quality data containing spam, noise, and duplicates [61,62].”
From A Survey of Large Language Models (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla stored a higher-precision copy of the weights in the sharded optimiser state.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example given the Gopher (Rae et al., 2021) compute budget of 5.76 × 10 23 FLOPs, their parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Small Gopher models also do not perform above random until scaled up to the largest model of 5 · 10 23 training FLOPs (280B parameters), for which performance jumps to more than 20% above random (Rae et al., 2021).”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For GPT-3, Gopher, and Chinchilla, models of ∼ 10 22 training FLOPs ( ∼ 10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to 3-5 · 10 23 training FLOPs (70B-280B parameters) enables performance to substantially surpass random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture (Hoffmann et al. 2022; Rae et al. 2021).”
From Algorithmic progress in language models (2024) · cites this paper as Rae et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021). In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“in the analysis of mod- els trained on the same corpus in Rae et al. (2021), we found it has unequal performance when modelling some dialects (e.g., African American English).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“The same dataset is used as in Rae et al. (2021).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Notably, the 70-billion parameter Chinchilla model (Hoffmann et al., 2022) outperforms the 280-billion parameter Gopher model (Rae et al., 2021) while using a similar compute budget by being trained on four times more data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The TruthfulQA benchmark (Lin et al., 2021) showed that GPT-3 models were more likely to mimic human falsehoods as they got larger, though Rae et al. (2021) later showed on a multiple-choice version that scaling Gopher to 280B enabled emergent performance substantially better than random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“| GitHub | 0.53 | 0.47 |”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 1
- [10]10. O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-SNLI: Natural language inference with natural language explanations. arXiv preprint arXiv:1812.01193, 2018. - Y. T. Cao and H. Daumé. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecyle. Computational Linguistics, pages 1-47, 2021. - N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019. - N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models, 2021. - N. Chater. The search for simplicity: A fundamental cognitive principle? The Quarterly Journal of Experimental Psychology Section A, 52(2):273-302, 1999. - H. Chen, X. Liu, D. Yin, and J. Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-35, 2017. - T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015. - P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. - D. Coyle and A. Weller. 'Explaining' machine learning reveals policy challenges. Science, 368(6498): 1433-1434, 2020. · arXiv
Cited by
Sorted by citations
Items 301–336 of 336
HyperT5: Towards Compute-Efficient Korean Language Modeling
Park, D., Ka, S., Yoo, K. M., Lee, G., & Kang, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.40
Banaei, M., Bałazy, K., Kasymov, A., Lebret, R., Tabor, J., & Aberer, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.133
Emergent Inabilities? Inverse Scaling Over the Course of Pretraining
Michaelov, J. A., & Bergen, B. K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.973 · Cited by 4
DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework
Kim, K., & Lee, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.150
Scaled Down Lean BERT-like Language Models for Anaphora Resolution and Beyond
Labs, N., Bolshakov, V., Kolobov, R., Borisov, E., Mikhaylovskiy, N., & Mukhtarova, G. (2023) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2023-22-1032-1040
Deep neural network models of emotion understandingPreprint
Thornton, M. A. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/s2rbq
Chaikiatsri, P., & Rattanasopon, S. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/smkfu
Large Language Model for automobilePreprint
Ding, F. (2024) · doi:10.33774/coe-2024-cdlwg
A Large Financial Chat ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-lsc94
Lin, P.-Y., Chandra, E., & Hsu, J. Y.-J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31195 · Cited by 1
ProofsNavigator: A Bootstrap Base Method for Explainable Knowledge Reasoning
韦, 泽. (2024) · Computer Science and Application · doi:10.12677/csa.2024.147159
Advancing low resource information extraction and dialogue system using data efficient methods
Ding, B. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/179560
Understanding Creators' Acceptance of Content Reuse
Sinha, R., & Li, H. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681911
Second language learning of degree expressions: A computational approach
Cong, Y. (2024) · Natural language processing. · doi:10.1017/nlp.2024.56
A Maritime Document Knowledge Graph Construction Method Based on Conceptual Proximity Relations
Lin, Y., Yang, T., Shao, Y., Yuan, M., Hu, P., & Li, C. (2025) · Journal of Computer and Communications · doi:10.4236/jcc.2025.132005
CURE: A dataset for Clinical Understanding & Retrieval Evaluation
Sheikh, N. A., Marcos, D. B., Jousse, A.-L., Oladipo, A., Rousseau, O., & Lin, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737435 · Cited by 2
Rong, B., Gifford, A. T., Düzel, E., & Cichy, R. M. (2025) · eLife · doi:10.7554/elife.108915.1 · Cited by 1
A Catalog of Transformer Models
D'Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r595153
A Catalog of Transformer Models
D'Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r595154
A Catalog of Transformer Models
D’Souza, J., & Rabby, G. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r656111
A Catalog of Transformer Models
D'Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r385010
A Catalog of Transformer Models
D’Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r606164
A Catalog of Transformer Models
D’Souza, J., & Rabby, G. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r656113 · Cited by 1
A Catalog of Transformer Models
D'Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r585918
A Catalog of DeepMind's Large Language Models including their Seminal Chinchilla Work
D’Souza, J. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r609271
An electronic product carbon footprint dataset for question answering
Zhao, K., Chathoth, A. K., Balaji, B., & Lee, S. (2026) · Scientific Data · doi:10.1038/s41597-026-06544-5
Understanding Social Media Users' Perspectives for Content Reuse
Sinha, R., & Li, H. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3799454
DLRNA-BERTa: a transformer approach for predicting molecule-RNA binding affinities
Lobascio, P., Saeed, K., Khan, A., & Tanoli, Z. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag488
Artificial Intelligence as an Emergent Order
Berg, C. (2026) · Palgrave studies in classical liberalism · doi:10.1007/978-3-032-22310-4_3
Integrating AI and Knowledge Graphs for Advancing Georisk Assessment
Chambi, S. P. V., Osorio, E. E. C., Nzikou, M. M., & Otiniano-Zavala, A. (2026) · doi:10.59490/6ad293d76ee698573e4dd235
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · arXiv · doi:10.48550/arxiv.2405.17893
“Nevertheless, such LLMs fall short in addressing mathematical problems that involves arithmetic, commonsense, and symbolic reasoning – topics that may appear deceptively simple to humans (Rae et al., 2021).”
BackgroundA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationSolving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citation
Metadata
- Authors
- Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. V. D., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., d'Autume, C. D. M., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., Casas, D. D. L., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., & Irving, G.
- ORCID
- Hendricks, Lisa Anne
- Rauh, Maribeth
- Huang, Po-Sen
- Dathathri, Sumanth
- Huang, Saffron
- Higgins, Irina
- Creswell, Antonia
- Budden, David
- Simonyan, Karen
- Paganini, Michela
- Lazaridou, Angeliki
- Mensch, Arthur
- Li, Yujia
- Mikulik, Vladimir
- Babuschkin, Igor
- Johnson, Matthew
- Weidinger, Laura
- Gabriel, Iason
- Vinyals, Oriol
- Ayoub, Kareem
- Hassabis, Demis
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.11446
- OpenAlex
- W4226146865
- Cited by
- 235
- References
- 0
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 52 · 2023: 121 · 2022: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- gophers, large language models, model scaling, transformer models, reading comprehension, fact-checking, toxic language detection, bias in language models, AI safety, mathematical reasoning, training data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
