CoChat
Sign Up
PreprintCited by 33,402Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

The DOI matches its DataCite record; 124 of the 127 references it cites were found in a registry or an index; none of the 48 sources its claims cite is checked for support yet.

Language Models are Few-Shot Learners

Open access
Brown, Tom B., Mann, Benjamin, Ryder, Nick, Subbiah, Melanie, Kaplan, Jared, Dhariwal, Prafulla, Neelakantan, Arvind, Shyam, Pranav, Sastry, Girish, Askell, Amanda, Agarwal, Sandhini, Herbert-Voss, Ariel, Krueger, Gretchen, Henighan, Tom, Child, Rewon, Ramesh, Aditya, Ziegler, Daniel M., Wu, Jeffrey, Winter, Clemens, Hesse, Christopher, Chen, Mark, Sigler, Eric, Litwin, Mateusz, Gray, Scott, Chess, Benjamin, Clark, Jack, Berner, Christopher, McCandlish, Sam, Radford, Alec, Sutskever, Ilya, Amodei, Dario·2020·arXiv
DOI 10.48550/arxiv.2005.14165arXiv 2005.14165
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

GPT-3 is a 175B-parameter autoregressive language model evaluated without gradient updates or fine-tuning on over two dozen NLP datasets and several synthetic tasks in zero-, one-, and few-shot settings. It often approaches or sometimes exceeds fine-tuned state of the art in few-shot settings, while still struggling on some comparison, NLI, and reading-comprehension tasks.

Key findings

  • GPT-3 achieves strong few-shot results without fine-tuning: CoQA F1 81.5 zero-shot, 84.0 one-shot, 85.0 few-shot; TriviaQA accuracy 64.3 zero-shot, 68.0 one-shot, 71.2 few-shot, the last matching or exceeding fine-tuned closed-book/open-domain systems in the paper’s comparison.
  • On LAMBADA, GPT-3 reaches 76.2% zero-shot and 86.4% few-shot accuracy, an 18% increase over prior state of the art; on PIQA it reaches 82.8% few-shot, above the 79.4% fine-tuned SOTA reported.
  • GPT-3 shows on-the-fly reasoning: 100% few-shot accuracy on 2-digit addition, 98.9% on 2-digit subtraction, 80.4% on 3-digit addition, 94.2% on 3-digit subtraction, 29.2% on 2-digit multiplication, and 21.3% on one-digit composite operations.
  • Human evaluators had difficulty distinguishing GPT-3-generated news articles from human-written ones: mean accuracy was 52% for GPT-3 175B versus 86% for a deliberately bad control model on ~200-word articles, and 52% versus 88% on ~500-word articles.

Methods

  • The paper trains 8 autoregressive transformer models from 125M to 175B parameters, all for 300 billion tokens with a 2048-token context window; GPT-3 175B is the largest.
  • Evaluation covers zero-shot, one-shot, and few-shot settings with no gradient updates or fine-tuning; few-shot conditioning uses typically 10 to 100 examples, and tasks are specified via text interaction.
  • The paper evaluates on language modeling, cloze, closed-book QA, translation, Winograd-style tasks, commonsense reasoning, reading comprehension, SuperGLUE, NLI, and synthetic tasks, and includes a systematic contamination analysis with cleaned benchmark subsets.

Limitations

  • The paper states GPT-3 still has weaknesses in text synthesis, including semantic repetition, loss of coherence over long passages, self-contradiction, and non-sequiturs; it also struggles with “common sense physics” and does little better than chance on some comparison tasks such as WiC and ANLI.
  • The paper states structural limitations: it uses only autoregressive models, not bidirectional or denoising objectives, which may hurt fill-in-the-blank, comparison, and long-passage tasks; it also notes limits of the pretraining objective, poor pretraining sample efficiency, ambiguity about whether few-shot learning is truly from scratch, high inference cost, and bias inherited from training data.
  • The paper states a contamination limitation: a filtering bug left some benchmark overlap, retraining was infeasible, and the clean-subset analysis cannot guarantee that cleaned examples come from the same distribution as the original dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2005.14165

Cites

124 of 124 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 2 2: Total compute used during training.Based on the analysis in Scaling Laws For Neural Language Models [KMH + 20] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training. Methodology for these calculations can be found in Appendix D.
Figure 2 2: Total compute used during training.Based on the analysis in Scaling Laws For Neural Language Models [KMH + 20] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training. Methodology for these calculations can be found in Appendix D.From the PDF
Figure 3 1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in [KMH + 20] continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts.
Figure 3 1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in [KMH + 20] continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts.From the PDF
Figure 3 . 3 : Figure3.3: On TriviaQA GPT3's performance grows smoothly with model size, suggesting that language models continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make significant gains over zero-shot behavior, matching and exceeding the performance of the SOTA fine-tuned open-domain model, RAG[LPP + 20]
Figure 3 . 3 : Figure3.3: On TriviaQA GPT3's performance grows smoothly with model size, suggesting that language models continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make significant gains over zero-shot behavior, matching and exceeding the performance of the SOTA fine-tuned open-domain model, RAG[LPP + 20]From the PDF
Figure 3 . 4 : Figure3.4: Few-shot translation performance on 6 language pairs as model capacity increases. There is a consistent trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be stronger than translation from English.
Figure 3 . 4 : Figure3.4: Few-shot translation performance on 6 language pairs as model capacity increases. There is a consistent trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be stronger than translation from English.From the PDF
Figure 3 . 5 : Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales.Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B is competitive with a fine-tuned RoBERTA-large.
Figure 3 . 5 : Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales.Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B is competitive with a fine-tuned RoBERTA-large.From the PDF
Figure 3 . 6 : Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings. The largest model achieves a score on the development set in all three conditions that exceeds the best recorded score on the task.
Figure 3 . 6 : Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings. The largest model achieves a score on the development set in all three conditions that exceeds the best recorded score on the task.From the PDF
Figure 3 9: Performance of GPT-3 on ANLI Round 3. Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%). We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA. Results for ANLI rounds 1 and 2 are shown in the appendix.
Figure 3 9: Performance of GPT-3 on ANLI Round 3. Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%). We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA. Results for ANLI rounds 1 and 2 are shown in the appendix.From the PDF
Figure 3 . Figure 3.11: Few-shot performance on the five word scrambling tasks for different sizes of model. There is generally smooth improvement with model size although the random insertion task shows an upward slope of improvement with the 175B model solving the task the majority of the time. Scaling of one-shot and zero-shot performance is shown in the appendix. All tasks are done with K = 100.
Figure 3 . Figure 3.11: Few-shot performance on the five word scrambling tasks for different sizes of model. There is generally smooth improvement with model size although the random insertion task shows an upward slope of improvement with the 175B model solving the task the majority of the time. Scaling of one-shot and zero-shot performance is shown in the appendix. All tasks are done with K = 100.From the PDF
Figure 3 . Figure3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest model achieves 65% accuracy in the few-shot setting, and also demonstrates significant gains to in-context learning which are not present in smaller models.
Figure 3 . Figure3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest model achieves 65% accuracy in the few-shot setting, and also demonstrates significant gains to in-context learning which are not present in smaller models.From the PDF
Figure 3 . Figure 3.13: People's ability to identify whether news articles are model-generated (measured by the ratio of correct assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberatelybad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best fit is a power law with 95% confidence intervals.
Figure 3 . Figure 3.13: People's ability to identify whether news articles are model-generated (measured by the ratio of correct assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberatelybad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best fit is a power law with 95% confidence intervals.From the PDF
Figure 4 1: GPT-3 Training Curves We measure model performance during training on a deduplicated validation split of our training distribution. Though there is some gap between training and validation performance, the gap grows only minimally with model size and training time, suggesting that most of the gap comes from a difference in difficulty rather than overfitting.
Figure 4 1: GPT-3 Training Curves We measure model performance during training on a deduplicated validation split of our training distribution. Though there is some gap between training and validation performance, the gap grows only minimally with model size and training time, suggesting that most of the gap comes from a difference in difficulty rather than overfitting.From the PDF
Figure 4 . 2 : Figure4.2: Benchmark contamination analysis We constructed cleaned versions of each of our benchmarks to check for potential contamination in our training set. The x-axis is a conservative lower bound for how much of the dataset is known with high confidence to be clean, and the y-axis shows the difference in performance when evaluating only on the verified clean subset. Performance on most benchmarks changed negligibly, but some were flagged for further review. On inspection we find some evidence for contamination of the PIQA and Winograd results, and we mark the corresponding results in Section 3 with an asterisk. We find no evidence that other benchmarks are affected.
Figure 4 . 2 : Figure4.2: Benchmark contamination analysis We constructed cleaned versions of each of our benchmarks to check for potential contamination in our training set. The x-axis is a conservative lower bound for how much of the dataset is known with high confidence to be clean, and the y-axis shows the difference in performance when evaluating only on the verified clean subset. Performance on most benchmarks changed negligibly, but some were flagged for further review. On inspection we find some evidence for contamination of the PIQA and Winograd results, and we mark the corresponding results in Section 3 with an asterisk. We find no evidence that other benchmarks are affected.From the PDF
Figure 6 . 1 : Figure 6.1: Racial Sentiment Across Models
Figure 6 . 1 : Figure 6.1: Racial Sentiment Across ModelsFrom the PDF
Figure H. 1 : Figure H.1: All results for all SuperGLUE tasks.
Figure H. 1 : Figure H.1: All results for all SuperGLUE tasks.From the PDF
Figure H. 2 : Figure H.2: Results for SAT task.Figure H.3: All results for all Winograd tasks.
Figure H. 2 : Figure H.2: Results for SAT task.Figure H.3: All results for all Winograd tasks.From the PDF
Figure H. 3 : Figure H.2: Results for SAT task.Figure H.3: All results for all Winograd tasks.
Figure H. 3 : Figure H.2: Results for SAT task.Figure H.3: All results for all Winograd tasks.From the PDF
Figure H. 4 : Figure H.4: All results for all Arithmetic tasks.
Figure H. 4 : Figure H.4: All results for all Arithmetic tasks.From the PDF
Figure H. 5 : Figure H.5: All results for all Cloze and Completion tasks.
Figure H. 5 : Figure H.5: All results for all Cloze and Completion tasks.From the PDF
Figure H. 6 : Figure H.6: All results for all Common Sense Reasoning tasks.
Figure H. 6 : Figure H.6: All results for all Common Sense Reasoning tasks.From the PDF
Figure H. 7 : Figure H.7: All results for all QA tasks.
Figure H. 7 : Figure H.7: All results for all QA tasks.From the PDF
Figure H. 8 : Figure H.8: All results for all Reading Comprehension tasks.
Figure H. 8 : Figure H.8: All results for all Reading Comprehension tasks.From the PDF
Figure H. 9 : Figure H.9: All results for all ANLI rounds.
Figure H. 9 : Figure H.9: All results for all ANLI rounds.From the PDF
Figure H. 10 : Figure H.10: All results for all Scramble tasks.
Figure H. 10 : Figure H.10: All results for all Scramble tasks.From the PDF
Figure H. 11 : Figure H.11: All results for all Translation tasks.
Figure H. 11 : Figure H.11: All results for all Translation tasks.From the PDF
Image 25
From the PDF
Image 26
From the PDF
Image 27
From the PDF
Image 28
From the PDF
Image 29
From the PDF
Image 30
From the PDF
Image 31
From the PDF
Image 32
From the PDF

Verification

Claims the paper makes

155 Own result · 60 Own interpretation · 41 No basis given · 40 Sources not found · 29 Own method · 1 Not checked yet

Show all 161 claims

  • It may state two separate claims as one. 111
  • Probably not a substantial claim. 79
  • The quote only partly states this claim. 1
165 of 165

  • The paper relies on this without support.

    To achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.

    “to achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.”

  • The paper relies on this without support.

    Humans do not require large supervised datasets to learn most language tasks; a brief directive or a tiny number of demonstrations is often sufficient to enable a human to perform a new task to at least a reasonable degree of competence.

    “humans do not require large supervised datasets to learn most language tasks - a brief directive in natural language (e.g. 'please tell me if this sentence describes something happy or something sad') or at most a tiny number of demonstrations (e.g. 'here are two examples of people acting brave; please give a third example of bravery') is often sufficient to enable a human to perform a new task to”

  • The paper relies on this without support.

    Models have no access to the specific facts that article titles refer to or when the article was written.

    “Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written.”

  • The paper relies on this without support.

    The news article titles and subtitles used were not in the models' training data.

    “The articles we selected were not in the models' training data and the model outputs were formatted and selected programmatically to prevent human cherry-picking.”

  • The paper relies on this without support.

    CL, A1, and A2 are not bijective.

    “Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling.”

  • The paper relies on this without support.

    The forums and chat groups monitored are representative of how low- and mid-skill threat actors think and behave.

    “To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed.”

  • The paper relies on this without support.

    APTs do not typically discuss operations in the open.

    “Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about possible APT activity involving the use of language models.”

  • The paper relies on this without support.

    Professional threat analysts' assessments about APT activity involving language models are reliable.

    “we have consulted with professional threat analysts about possible APT activity involving the use of language models.”

  • The paper relies on this without support.

    The 388 occupations tested are representative of occupations in general.

    “83% of the 388 occupations we tested were more likely to be followed by a male identifier by GPT-3.”

  • The authors state this as an assumption.

    The attention operation typically uses less than 10% of the total compute for the models analyzed.

    “As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10% of the total compute for the models we are analyzing.”

  • The paper relies on this without support.

    T5 uses an encoder-decoder model, so only half of the parameters are active for each token during a forward or backwards pass.

    “Next we note that since T5 uses an encoder-decoder model, only half of the parameters are active for each token during a forward or backwards pass.”

  • The paper relies on this without support.

    The information required to answer DROP questions is in a passage provided to the model.

    “The information required to answer the question is in a passage provided to the model, so having seen the passage during training but not the questions and answers does not meaningfully constitute cheating.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 127 references

Cited by

Sorted by citations

Items 30,001–30,100 of 33,402

Related

See all

Metadata

Authors
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
1877-1901
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2005.14165
arXiv
2005.14165
OpenAlex
W3030163527
Semantic Scholar
90abbc2cf38462b954ae1b772fac9532e2ccd8b0
Cited by
23,372
References
127
Citations by year
2026: 403 · 2025: 4739 · 2024: 7286 · 2023: 6380 · 2022: 2283 · 2021: 1844 · 2020: 427
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
National Science Foundation (2112606)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
few-shot learning, GPT-3, large language models, task-agnostic learning, pre-training, fine-tuning, autoregressive language models, domain adaptation, text-to-text translation, question answering, cloze task, text generation, societal impact of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review