The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Open accessAbstract
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs are particularly vulnerable to performance degradation from low-quality data containing spam, noise, and duplicates [61,62].”
From A Survey of Large Language Models (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This limitation stems from the context window of LLMs, which today is limited to at most 2048 tokens (i.e. about 1500 words) in state-of-the-art models [78,86].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chinchilla stored a higher-precision copy of the weights in the sharded optimiser state.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example given the Gopher (Rae et al., 2021) compute budget of 5.76 × 10 23 FLOPs, their parametric loss function fitted on MassiveWeb predicts an optimal allocation of 40 billion parameters.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Small Gopher models also do not perform above random until scaled up to the largest model of 5 · 10 23 training FLOPs (280B parameters), for which performance jumps to more than 20% above random (Rae et al., 2021).”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For GPT-3, Gopher, and Chinchilla, models of ∼ 10 22 training FLOPs ( ∼ 10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to 3-5 · 10 23 training FLOPs (70B-280B parameters) enables performance to substantially surpass random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling law from which our model is derived applies to dense transformers following a GPT-3 architecture (Hoffmann et al. 2022; Rae et al. 2021).”
From Algorithmic progress in language models (2024) · cites this paper as Rae et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021). In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“in the analysis of mod- els trained on the same corpus in Rae et al. (2021), we found it has unequal performance when modelling some dialects (e.g., African American English).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“The same dataset is used as in Rae et al. (2021).”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Notably, the 70-billion parameter Chinchilla model (Hoffmann et al., 2022) outperforms the 280-billion parameter Gopher model (Rae et al., 2021) while using a similar compute budget by being trained on four times more data.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2022) · cites this paper as Rae et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“The TruthfulQA benchmark (Lin et al., 2021) showed that GPT-3 models were more likely to mimic human falsehoods as they got larger, though Rae et al. (2021) later showed on a multiple-choice version that scaling Gopher to 280B enabled emergent performance substantially better than random.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Rae et al., 2021
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“| GitHub | 0.53 | 0.47 |”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Rae et al., 2021
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Rae et al. (2021)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 1
- [10]10. O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-SNLI: Natural language inference with natural language explanations. arXiv preprint arXiv:1812.01193, 2018. - Y. T. Cao and H. Daumé. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecyle. Computational Linguistics, pages 1-47, 2021. - N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019. - N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models, 2021. - N. Chater. The search for simplicity: A fundamental cognitive principle? The Quarterly Journal of Experimental Psychology Section A, 52(2):273-302, 1999. - H. Chen, X. Liu, D. Yin, and J. Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-35, 2017. - T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015. - P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. - D. Coyle and A. Weller. 'Explaining' machine learning reveals policy challenges. Science, 368(6498): 1433-1434, 2020. · arXiv
Cited by
Sorted by citations
Items 101–200 of 336
VisualWebArena: evaluating multimodal agents on realistic visual web tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.50 · Cited by 50
Self-citationA Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 56
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Gururangan, S., Card, D., Dreier, S. K., Gade, E. K., Wang, L. Z., Wang, Z., Zettlemoyer, L., & Smith, N. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.165 · Cited by 87
Artificial intelligence generated content (AIGC) in medicine: A narrative review
Shao, L., Chen, B., Zhang, Z., Zhang, Z., & Chen, X. (2024) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2024073 · Cited by 51
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 54
Enhanced automated code vulnerability repair using large language models
de-Fitero-Dominguez, D., Garcia-Lopez, E., Garcia-Cabot, A., & Martinez-Herraiz, J.-J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109291 · Cited by 43
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 91
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
Kauf, C., Tuckute, G., Lévy, R., Andreas, J., & Fedorenko, E. G. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00116 · Cited by 48
Almanac: Retrieval-Augmented Language Models for Clinical MedicinePreprint
Hiesinger, W., Zakka, C., Chaurasia, A., Shad, R., Dalal, A. R., Kim, J. L., Moor, M., Alexander, K. M., Ashley, E. A., Boyd, J. H., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2883198/v1 · Cited by 40
Pan, X., Li, X., Li, Q., Hu, Z., & Bao, J. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2023.2301230 · Cited by 40
Self-citationOn the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.380 · Cited by 43
Enhancing Accuracy in Large Language Models Through Dynamic Real-Time Information InjectionPreprint
Ouyang, Q., Wang, S., & Wang, B. (2023) · Preprints.org · doi:10.20944/preprints202312.1987.v1 · Cited by 36
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2312.08361 · Cited by 35
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Kazemi, M., Kim, N., Bhatia, D. S., Xu, X., & Ramachandran, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.361 · Cited by 79
Belghith, Y., Goloujeh, A. M., Magerko, B. S., Long, D., McKlin, T., & Roberts, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642332 · Cited by 45
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
TheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 141
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 39
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 74
Aligning Generative Language Models with Human Values
Liu, R., Zhang, G., Feng, X., & Vosoughi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.18 · Cited by 37
Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Fu, L., Newman, B., Jakesch, M., & Kreps, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581351 · Cited by 31
Chen, H.-T., Zhang, M., & Choi, E. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.146 · Cited by 31
Chen, H.-T., Zhang, M. J. Q., & Choi, E. (2022) · arXiv · doi:10.48550/arxiv.2210.13701 · Cited by 31
AI Writing Assistants Influence Topic Choice in Self-Presentation
Poddar, R., Sinha, R., Naaman, M., & Jakesch, M. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585893 · Cited by 30
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization
Ladhak, F., Durmus, E., Suzgun, M., Zhang, T., Jurafsky, D., McKeown, K., & Hashimoto, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.234 · Cited by 30
Lai, V. D., Nguyen, C., Ngo, N., Nguyen, T., Dernoncourt, F., Rossi, R. A., & Nguyen, T. H. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.28 · Cited by 51
Generating Natural Language Proofs with Verifier-Guided Search
Yang, K., Deng, J., & Chen, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.7 · Cited by 55
The language of prompting: What linguistic properties make a prompt successful?
Leidinger, A., van Rooij, R., & Shutova, E. (2023) · arXiv · doi:10.48550/arxiv.2311.01967 · Cited by 29
Yang, H., Hu, M., Most, A. K., Hawkins, W. A., Murray, B., Smith, S. E., Li, S., & Sikora, A. N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1514896 · Cited by 28
Better Zero-Shot Reasoning with Self-Adaptive Prompting
Wan, X., Sun, R., Dai, H., Arık, S. Ö., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.216 · Cited by 56
AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs.
Ghosh, M., Mukherjee, S., Ganguly, A., Basuchowdhuri, P., Naskar, S. K., & Ganguly, D. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.005 · Cited by 27
Scaling Law for Recommendation Models: Towards General-Purpose User Representations
Shin, K., Kwak, H., Kim, S. Y., Ramström, M. N., Jeong, J.-S., Ha, J.-W., & Kim, K.-M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25582 · Cited by 43
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 95
Analyzing and Mitigating Cultural Hallucinations of Commercial Language Models in TurkishPreprint
Boztemir, Y., & Çalışkan, N. (2024) · doi:10.36227/techrxiv.171504678.80895811/v1 · Cited by 26
Inverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25
Leveraging Large Language Models to Generate Answer Set Programs
Ishay, A., Yang, Z., & Lee, J. (2023) · Proceedings of the Twentieth International Conference on Principles of Knowledge Representation and Reasoning · doi:10.24963/kr.2023/37 · Cited by 25
Nonparametric Masked Language Modeling
Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-T., Hajishirzi, H., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.132 · Cited by 50
Controllable Dialogue Simulation with In-context Learning
Li, Z., Chen, W., Li, S., Wang, H., Jing, Q., & Yan, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.318 · Cited by 40
Testolin, A. (2024) · Applied Sciences · doi:10.3390/app14020744 · Cited by 24
Espejel, J. L., Alassan, M. S. Y., Chouham, E. M., Dahhane, W., & Ettifouri, E. H. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100013 · Cited by 26
Lin, H., Luo, Z., Ma, J., & Chen, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.611 · Cited by 40
Wong, S. M., Leung, H.-F., & Wong, K. Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4063228/v1 · Cited by 23
Efficient Training and Inference: Techniques for Large Language Models Using LlamaPreprint
Cunningham, S. R., Archambault, D., & Kung, A. (2024) · doi:10.36227/techrxiv.171651876.65094225/v1 · Cited by 22
Xiao, C., Cai, J., Zhao, W., Lin, B., Zeng, G., Zhou, J., Zheng, Z., Han, X., Liu, Z., & Sun, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01137-0 · Cited by 24
“Guided by this trend, PaLM 540B 10 , Gopher-280B 47 and Llama-3-405B 11 achieve great improvements on various natural language processing tasks.”
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Wang, B., Ping, W., Xu, P., McAfee, L., Liu, Z., Shoeybi, M., Dong, Y., Kuchaiev, O., Li, B., Xiao, C., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.482 · Cited by 21
TextDescriptives: A Python package for calculating a large variety of metrics from text
Hansen, L., Olsen, L. R., & Enevoldsen, K. (2023) · The Journal of Open Source Software · doi:10.21105/joss.05153 · Cited by 21
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19
“…while we make use of a similar Transformer-XL architecture (Vaswani et al., 2017; Dai et al., 2019), we demonstrate scaling laws for online meta-RL that resemble those seen in other communities, such as language (Devlin et al., 2019; Kaplan et al., 2020; Brown et al., 2020; Rae et al., 2021).”
MethodFairness-Aware Structured Pruning in Transformers
Zayed, A., Mordido, G., Shabanian, S., Baldini, I., & Chandar, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30256 · Cited by 25
Evaluation of Transfer Learning and Adaptability in Large Language Models with the GLUE BenchmarkPreprint
Sulaiman, N., & Hamzah, F. (2024) · doi:10.36227/techrxiv.171077989.99407624/v1 · Cited by 20
Horawalavithana, S., Ayton, E., Sharma, S. K., Howland, S., Subramanian, M., Vasquez, S., Cosbey, R., Glenski, M., & Volkova, S. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.12 · Cited by 20
Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification
Li, S., Zhao, R., Li, M., Ji, H., Callison-Burch, C., & Han, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.312 · Cited by 29
How Do In-Context Examples Affect Compositional Generalization?
An, S., Lin, Z., Fu, Q., Chen, B., Zheng, N., Lou, J., & Zhang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.618 · Cited by 46
Bioeconomy firms and where to find them
Kriesch, L., & Losacker, S. (2024) · REGION · doi:10.18335/region.v11i1.523 · Cited by 19
A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
Lakim, I., Almazrouei, E., Abualhaol, I. Y., Debbah, M. A., & Launay, J. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.8 · Cited by 19
Wang, S., Ouyang, Q., & Wang, B. (2024) · Research Square · doi:10.21203/rs.3.rs-3987793/v1 · Cited by 20
A Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mc762 · Cited by 23
Goto, T., Ono, K., & Morita, A. (2024) · doi:10.36227/techrxiv.171173447.70655950/v1 · Cited by 19
Measuring the Interpretability and Explainability of Model Decisions of Five Large Language ModelsPreprint
Fujiwara, K., Sasaki, M., Nakamura, A., & Watanabe, N. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/d4ntw · Cited by 20
Deep Lexical Hypothesis: Identifying personality structure in natural languagePreprint
Cutler, A., & Condon, D. M. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/gdm5v · Cited by 19
Efficient Compression of Large Language Models: A Case Study on Llama 2 with 13B ParametersPreprint
Konishi, M., Nakano, K., & Tomoda, Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4407954/v1 · Cited by 19
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
Making Large Language Models Better Data Creators
Lee, D., Pujara, J., Sewak, M., White, R. W., & Jauhar, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.948 · Cited by 28
Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing
Denny, P., Sarsa, S., Hellas, A., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.04715 · Cited by 21
Knowledge Distillation of Russian Language Models with Reduction of Vocabulary
Kolesnikova, A., Kuratov, Y., Konovalov, V., & Mikhail, M. (2022) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2022-21-295-310 · Cited by 23
Randomized Positional Encodings Boost Length Generalization of Transformers
Ruoss, A., Delétang, G., Genewein, T., Grau-Moya, J., Csordás, R., Bennani, M., Legg, S., & Veness, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.161 · Cited by 93
To what extent does ChatGPT understand genetics?
Khosravi, T., Sudani, Z. M. A., & Oladnabi, M. (2023) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2258842 · Cited by 26
Skill-Based Few-Shot Selection for In-Context Learning
An, S., Zhou, B., Lin, Z., Fu, Q., Chen, B., Zheng, N., Chen, W., & Lou, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.831 · Cited by 42
Human-like property induction is a challenge for large language modelsPreprint
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2022) · doi:10.31234/osf.io/6mkjy · Cited by 16
Integrating Deep Learning with Symbolic Reasoning in TinyLlama for Accurate Information RetrievalPreprint
Xiong, X., & Zheng, M. (2024) · Research Square · doi:10.21203/rs.3.rs-3883562/v1 · Cited by 18
Algorithmic progress in language models
Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., & Sevilla, J. (2024) · arXiv · doi:10.48550/arxiv.2403.05812 · Cited by 16
In conversation with Artificial Intelligence: aligning language models with human values
Kasirzadeh, A., & Gabriel, I. (2022) · arXiv · doi:10.48550/arxiv.2209.00731 · Cited by 128
“2For GPT-3, see [19]; for BERT, see [28]; for Turing-NLG-A-17, see [85]; for CLIP, see [81]; for Gopher, see [82].”
BackgroundSelf-citationShould ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Era
Zhang, D. (2023) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2023.1163 · Cited by 19
Data Curation Alone Can Stabilize In-context Learning
Chang, T.-Y., & Jia, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.452 · Cited by 37
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Shamardina, T., Mikhailov, V., Chernianskii, D., Fenogenova, A., Saidov, M., Valeeva, A., Shavrina, T., Smurov, I., Tutubalina, E. V., & Artemova, E. (2022) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2022-21-497-511 · Cited by 34
Enhancing Contextual Understanding of Mistral LLM with External Knowledge BasesPreprint
Sasaki, M., Watanabe, N., & Komanaka, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4215447/v1 · Cited by 17
In-context Reinforcement Learning with Algorithm Distillation
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14215 · Cited by 14
“For example, large language models like GPT-3 (Brown et al., 2020) or Gopher (Rae et al., 2021) can be directed at solving tasks such as text completion, code generation, and text summarization by specifying the task through language as a prompt.”
BackgroundMichaelov, J. A., & Bergen, B. K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.891 · Cited by 16
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14
JASMINE: Arabic GPT Models for Few-Shot Learning
Nagoudi, E. M. B., Abdul-Mageed, M., Elmadany, A., Inciarte, A. A., & Khondaker, M. T. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1040 · Cited by 25
Foundation Models in Healthcare: Opportunities, Biases and Regulatory Prospects in Europe
Wójcik, M. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12673-4_3 · Cited by 14
Training Trajectories of Language Models Across Scales
Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., & Stoyanov, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.767 · Cited by 49
MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussions
Sun, H., Zhang, Z., Mi, F., Wang, Y., Liu, W., Cui, J., Wang, B., Liu, Q., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.123 · Cited by 22
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models
Li, C., Chen, H., Yan, M., Shen, W., Xu, H., Wu, Z., Zhang, Z., Zhou, W., Chen, Y., Chen, C., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.51 · Cited by 14
Wang, Y., Cai, Y., Chen, M., Liang, Y., & Hooi, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.8 · Cited by 48
Using In-Context Learning to Improve Dialogue Safety
Meade, N., Gella, S., Hazarika, D., Gupta, P., Jin, D., Reddy, S., Liu, Y., & Hakkani-Tur, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.796 · Cited by 13
OpenICL: An Open-Source Framework for In-context Learning
Wu, Z., Wang, Y., Ye, J., Wu, Z., Feng, J., Xu, J., & Qiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.47 · Cited by 14
Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., & Roth, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.660 · Cited by 47
Kwon, S. J., Kim, J., Bae, J., Yoo, K. M., Kim, J.-H., Park, B., Kim, B., Ha, J.-W., Sung, N., & Lee, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.240 · Cited by 17
Toward robust natural language systems
Moon, H. C. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/169803 · Cited by 13
GPT-Neo-CRV: Elevating Information Accuracy in GPT-Neo with Cross-Referential ValidationPreprint
Xiong, X., & Zheng, M. (2024) · doi:10.36227/techrxiv.170473976.61241065/v1 · Cited by 13
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.61 · Cited by 12
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models
Xia, M., Artetxe, M., Du, J., Chen, D., & Stoyanov, V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.780 · Cited by 15
Are Emergent Abilities in Large Language Models just In-Context Learning?
Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12
Transformers learn to implement preconditioned gradient descent for in-context learning
Ahn, K., Cheng, X., Daneshmand, H., & Sra, S. (2023) · arXiv · doi:10.48550/arxiv.2306.00297 · Cited by 11
“…striking capability of large language models: Given a prompt containing examples and a query, the transformer produces the correct output based on the context provided by the examples, without adapting its parameters (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Black et al., 2022).”
Background
Metadata
- Authors
- Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. V. D., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., d'Autume, C. D. M., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., Casas, D. D. L., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., & Irving, G.
- ORCID
- Hendricks, Lisa Anne
- Rauh, Maribeth
- Huang, Po-Sen
- Dathathri, Sumanth
- Huang, Saffron
- Higgins, Irina
- Creswell, Antonia
- Budden, David
- Simonyan, Karen
- Paganini, Michela
- Lazaridou, Angeliki
- Mensch, Arthur
- Li, Yujia
- Mikulik, Vladimir
- Babuschkin, Igor
- Johnson, Matthew
- Weidinger, Laura
- Gabriel, Iason
- Vinyals, Oriol
- Ayoub, Kareem
- Hassabis, Demis
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.11446
- OpenAlex
- W4226146865
- Cited by
- 235
- References
- 0
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 52 · 2023: 121 · 2022: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- gophers, large language models, model scaling, transformer models, reading comprehension, fact-checking, toxic language detection, bias in language models, AI safety, mathematical reasoning, training data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
