The DOI matches its DataCite record; 124 of the 127 references it cites were found in a registry or an index; none of the 48 sources its claims cite is checked for support yet.
Language Models are Few-Shot Learners
Open accessAbstract
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
GPT-3 is a 175B-parameter autoregressive language model evaluated without gradient updates or fine-tuning on over two dozen NLP datasets and several synthetic tasks in zero-, one-, and few-shot settings. It often approaches or sometimes exceeds fine-tuned state of the art in few-shot settings, while still struggling on some comparison, NLI, and reading-comprehension tasks.
Key findings
- GPT-3 achieves strong few-shot results without fine-tuning: CoQA F1 81.5 zero-shot, 84.0 one-shot, 85.0 few-shot; TriviaQA accuracy 64.3 zero-shot, 68.0 one-shot, 71.2 few-shot, the last matching or exceeding fine-tuned closed-book/open-domain systems in the paper’s comparison.
- On LAMBADA, GPT-3 reaches 76.2% zero-shot and 86.4% few-shot accuracy, an 18% increase over prior state of the art; on PIQA it reaches 82.8% few-shot, above the 79.4% fine-tuned SOTA reported.
- GPT-3 shows on-the-fly reasoning: 100% few-shot accuracy on 2-digit addition, 98.9% on 2-digit subtraction, 80.4% on 3-digit addition, 94.2% on 3-digit subtraction, 29.2% on 2-digit multiplication, and 21.3% on one-digit composite operations.
- Human evaluators had difficulty distinguishing GPT-3-generated news articles from human-written ones: mean accuracy was 52% for GPT-3 175B versus 86% for a deliberately bad control model on ~200-word articles, and 52% versus 88% on ~500-word articles.
Methods
- The paper trains 8 autoregressive transformer models from 125M to 175B parameters, all for 300 billion tokens with a 2048-token context window; GPT-3 175B is the largest.
- Evaluation covers zero-shot, one-shot, and few-shot settings with no gradient updates or fine-tuning; few-shot conditioning uses typically 10 to 100 examples, and tasks are specified via text interaction.
- The paper evaluates on language modeling, cloze, closed-book QA, translation, Winograd-style tasks, commonsense reasoning, reading comprehension, SuperGLUE, NLI, and synthetic tasks, and includes a systematic contamination analysis with cleaned benchmark subsets.
Limitations
- The paper states GPT-3 still has weaknesses in text synthesis, including semantic repetition, loss of coherence over long passages, self-contradiction, and non-sequiturs; it also struggles with “common sense physics” and does little better than chance on some comparison tasks such as WiC and ANLI.
- The paper states structural limitations: it uses only autoregressive models, not bidirectional or denoising objectives, which may hurt fill-in-the-blank, comparison, and long-passage tasks; it also notes limits of the pretraining objective, poor pretraining sample efficiency, ambiguity about whether few-shot learning is truly from scratch, high inference cost, and bias inherited from training data.
- The paper states a contamination limitation: a filtering bug left some benchmark overlap, retraining was infeasible, and the clean-subset analysis cannot guarantee that cleaned examples come from the same distribution as the original dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 2 2: Total compute used during training.Based on the analysis in Scaling Laws For Neural Language Models [KMH + 20] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training. Methodology for these calculations can be found in Appendix D.](/paper-figures/7000d92a5af643ffb355a5d30e5ea9b3/language-models-are-few-shot-learners-figure-2.png)
![Figure 3 1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in [KMH + 20] continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts.](/paper-figures/de2658e3e6e8418992de5f87a1b16ba2/language-models-are-few-shot-learners-figure-3.png)
![Figure 3 . 3 : Figure3.3: On TriviaQA GPT3's performance grows smoothly with model size, suggesting that language models continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make significant gains over zero-shot behavior, matching and exceeding the performance of the SOTA fine-tuned open-domain model, RAG[LPP + 20]](/paper-figures/53dcbf92e02540cdbb05cf999d3eba74/language-models-are-few-shot-learners-figure-3.png)
Verification
Claims the paper makes
155 Own result · 60 Own interpretation · 41 No basis given · 40 Sources not found · 29 Own method · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we have found that unfiltered or lightly filtered versions of Common Crawl tend to have lower quality than more curated datasets.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Broadly, for most tasks we find relatively smooth scaling with model capacity in all three settings; one notable pattern is that the gap between zero-, one-, and few-shot performance often grows with model capacity, perhaps suggesting that larger models are more proficient meta-learners.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3's few-shot result further improves performance another 3.2% beyond this.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, we also find some tasks on which few-shot performance struggles, even at the scale of GPT-3. This includes natural language inference tasks like the ANLI dataset, and some reading comprehension datasets like RACE or QuAC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On CB, we see signs of life at 75.6% in the few-shot setting.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On WSC, performance is still relatively strong, achieving 80.1% in the few-shot setting”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of unsupervised pretraining, supervised finetuning on 608K labeled examples, and backtranslation [LHCG19b].”
What the sources say
- Not matchedcites LHCG19b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though the results in this case are particularly striking, the general trends with both model size and number of examples in-context hold for most tasks we study.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 111
- Probably not a substantial claim. 79
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 has a noticeable skew in its performance depending on language direction. For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA's test server).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On the 'Easy' version of the dataset (questions which either of the mentioned baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a fine-tuned RoBERTa baseline from [KKS + 20].”
What the sources say
- Not matchedcites KKS + 20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although we find that data contamination has a minimal effect on GPT-3's performance on most datasets, we do identify a few datasets where it could be inflating results, and we either do not report results on these datasets or we note them with an asterisk, depending on the severity.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3's few-shot results by 27% on the challenge set and 22% on the easy set.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [LDL19] but improves over previous zero-shot results by roughly 10%.”
What the sources say
- Not matchedcites LDL19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, on one of the three datasets GPT-3's one-shot matches the open-domain fine-tuning SOTA. On the other two datasets it approaches the performance of the closed-book SOTA despite not using fine-tuning.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a significant improvement is observed on OpenBookQA.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase of over 18% from the previous state-of-the-art.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“On DROP [DWD + 19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL + 19].”
What the sources say
- Not matchedcites DWD + 19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites RLL + 19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [ZHR + 19] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM.”
What the sources say
- Not matchedcites ZHR + 19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Model performance improves with the addition of a natural language task description, and with the number of examples in the model's context, K .”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both Fr-En and De-En, few shot GPT-3 outperforms the best supervised result we could find”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although GPT-3's training data is still primarily English (93% by word count), it also includes 7% of text in other languages.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion parameter model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held by fine-tuned models).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to fine-tuned models operating in the same closed-book setting.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with K = 70 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where first place is held by a fine-tuned 11 billion parameter model (T5).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the 'Challenge' version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot setting, and 51.5% in the few-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, providing only a single example demonstration for each translation task improves performance by over 7 BLEU and nears competitive performance with prior work.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The one-shot result improves by 3.7% and matches the SOTA for an open-domain QA system which not only fine-tunes but also makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [LPP + 20].”
What the sources say
- Not matchedcites LPP + 20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“On OpenBookQA [MCKS18], GPT-3 improves significantly from zero to few shot settings but is still over 20 points short of the overall SOTA.”
What the sources say
- Not matchedcites MCKS18
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper.”
What the sources say
- Not matchedcites RJL18
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Larger values of K are usually but not always better, so when a separate development and test set are available, we experiment with a few values of K on the development set and then run the best value on the test set.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI + 18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions.”
What the sources say
- Not matchedcites RCM19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites CHI + 18
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“On RACE [LXL + 17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.”
What the sources say
- Not matchedcites LXL + 17
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average occupation bias - measured as 1 n jobs ∑ jobs log( P (female | Context) P (male | Context)) ) - was -1 . 11 for the Neutral Variant, -2 . 14 for the Competent Variant and -1 . 15 for the Incompetent Variant.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a significant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“as well as on a subset of reading comprehension tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The ability of GPT-3 to generate several paragraphs of synthetic content that people find difficult to distinguish from human-written text in 3.9.4 represents a concerning milestone in this regard.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-3 samples still sometimes repeat themselves semantically at the document level”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that words such as violent , terrorism and terrorist co-occurred at a greater rate with Islam than with other religions and were in the top 40 most favored words for Islam in GPT-3.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“it retains the biases of the data it has been trained on”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While we did find significant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances of experimentation and no successful deployments since then.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the final sentence the model generates a plausible conjugation for the word 'screeg' (namely 'screeghed'), although the use of the word is slightly awkward ('screeghed at each other') despite being plausible in the sense that it could describe a toy sword fight.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, 'Black' had a consistently low sentiment - it ranked the lowest in 5 out of 7 models.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language modeling benchmark.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“training the GPT-3 175B consumed several thousand petaflop/s-days of compute during pre-training, compared to tens of petaflop/s-days for a 1.5B parameter GPT-2 model (Figure 2.2).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The overlap analysis flagged 29% of examples as contaminated, and observed a 3 percentage point absolute decrease (4% relative decrease) in performance on the clean subset.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“contradict themselves”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“its decisions are not easily interpretable”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting ( ∼ 33% ), whereas GPT-3 itself shows signs of life on Round 3.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k truncation) they are not able to perform consistently without human feedback.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One notable exception was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very small, with the clean subset scoring within 0.5% of the full dataset.”
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“occasionally contain non-sequitur sentences or paragraphs”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“even with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or only a few cents in energy costs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Another limitation broadly shared by language models is poor sample efficiency during pre-training.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reflect stereotypes present in their training data.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Zero-shot and one-shot performance is a few points behind, with the gains to few-shot being largest for bigger models.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Related to this, the symbol insertion task shows high overlap but no effect on performance - this is because that task involves removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to many spurious matches.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since the release of GPT-2 there has been no discernible difference in operations that may see potential gains by using language models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: APTs do not typically discuss operations in the open.
- Rests on an assumption: Professional threat analysts' assessments about APT activity involving language models are reliable.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found 25% of the examples in the WMT16 German-English test set were marked as potentially contaminated, with an associated total effect size of 1-2 BLEU.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the models we analyzed, 'Asian' had a consistently high sentiment - it ranked 1st in 3 out of 7 models.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“we have noticed informally that GPT-3 seems to have special difficulty with 'common sense physics', despite doing well on some datasets (such as PIQA [BZB + 19]) that test this domain.”
What the sources say
- Not matchedcites BZB + 19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and sheriff.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-3 improves the quality of text generation and adaptability over smaller models and increases the difficulty of distinguishing synthetic text from human-written text.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“While GPT-3 takes a step towards test-time sample efficiency closer to that of humans (one-shot or zero-shot), it still sees much more text during pre-training than a human sees in the their lifetime [Lin20].”
What the sources say
- Not matchedcites Lin20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The overlap analysis flagged 45% of examples, and found a 2.6% decrease in performance on the clean subset.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Upon inspection, none of the flagged examples contain paired sentences resembling NMT training data and collisions were monolingual matches mostly of snippets of events discussed in the news.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Mean participant age was ∼ 38 years old.”
hedgedn = 621- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We confirmed that every matching training document contained only the source passage, and none of the questions and answers in the dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but there is no apparent bias towards improved or degraded performance.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Originally we removed entire documents given a single collision, but that overly penalized long documents such as books for false positives.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the curve can perhaps be detected in Figure 3.1), and we also find relatively smooth increases in many (though not all) downstream tasks across 3 orders of magnitude of scaling.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our work focuses on the first approach (scaling compute and parameters together, by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ this strategy.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall this decreased dataset size by an average of 10%.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of state-of-the-art fine-tuned systems, as well as generating high-quality samples and strong qualitative performance at tasks defined on-the-fly.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 175B | 3.64E+03 | 3.14E+23 | 174,600 | 300 | 6 | 3 | 2 | 1.0 |”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The attention operation typically uses less than 10% of the total compute for the models analyzed.
- Rests on an assumption: T5 uses an encoder-decoder model, so only half of the parameters are active for each token during a forward or backwards pass.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The main disadvantage is that results from this method have so far been much worse than state-of-the-art fine-tuned models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could also explain the comparatively low scores for RTE and CB, which also follow this format.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a smooth trend of improvement with scale [KMH + 20].”
What the sources say
- Not matchedcites [KMH + 20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“As found in [KMH + 20, MKAT18], larger models can typically use a larger batch size, but require a smaller learning rate.”
What the sources say
- Not matchedcites [KMH + 20, MKAT18]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“for example [RWC + 19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind the state of the art.”
What the sources say
- Not matchedcites [RWC + 19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4).”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions and/or the style of their answers are out-of-distribution for GPT-3.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Previous work [KMH + 20] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks.”
What the sources say
- Not matchedcites [KMH + 20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates an association- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- Not matched
In the paper
“For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [SBBC19] is 94.0%.”
What the sources say
- Not matchedcites SBBC19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to WebQS, the large gain from zero-shot to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to TriviaQA and WebQS.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“perhaps suggesting that larger models are more proficient meta-learners.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The main advantages of few-shot are a major reduction in the need for task-specific data and reduced potential to learn an overly narrow distribution from a large but narrow fine-tuning dataset.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: To achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, the questions in NQs tend towards very fine-grained knowledge on Wikipedia specifically which could be testing the limits of GPT-3's capacity and broad pretraining distribution.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Previous work [KMH + 20] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range.”
What the sources say
- Not matchedcites [KMH + 20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Recent work [RWC + 19] attempts to do this via what we call 'in-context learning', using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.”
What the sources say
- Not matchedcites [RWC + 19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of models of this scale in their current form.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Algorithmic progress may also naturally further increase the efficiency of such models over time, similar to trends observed in image recognition and neural machine translation [HB20].”
What the sources say
- Not matchedcites HB20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found a similar decrease in a 25x smaller model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias rather than memorization; examples which workers copied may simply be easier.”
hedgedstates a possible cause- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Related work on language model detection by Ippolito et al. [IDCBE19] indicates that automatic discriminators like GROVER [ZHR + 19] and GLTR [GSR19] may have greater success at detecting model generated text than human evaluators.”
What the sources say
- Not matchedcites [IDCBE19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [ZHR + 19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [GSR19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Where GPT-3 is on this spectrum may also vary from task to task.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Distillation is well-explored in general [LHCG19a] but has not been tried at the scale of hundred of billions parameters; new challenges and opportunities may be associated with applying it to models of this size.”
What the sources say
- Not matchedcites LHCG19a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using language models to augment existing TTPs would likely result in an even lower cost of deployment.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a social media disinformation bot produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the amount of human labor required in operating this bot. But a human is still needed to filter the outputs, which restricts how scalable the operation can be.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a preliminary analysis and does not reflect all of the model's biases even within the studied categories.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of these applications bottleneck on human beings to write sufficiently high quality text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Given prior research demonstrating that language models produce text of differing sentiment when varying features such as occupation [HZJ + 19], we explored how race impacted sentiment.”
What the sources say
- Not matchedcites HZJ + 19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks 'from scratch' at inference time, or if it simply recognizes and identifies tasks that it has learned during training.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Models have no access to the specific facts that article titles refer to or when the article was written.
In the paper
“This is concerning, since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and producing demeaning portrayals amongst other potential harms [Cra17].”
What the sources say
- Not matchedcites Cra17
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“On this task GPT-3 achieves 65.2% in the few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among college applicants was 57% [TL05] (random guessing yields 20%).”
What the sources say
- Not matchedcites [TL05]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This may include fill-in-the-blank tasks, tasks that involve looking back and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then generating a very short answer.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Additionally, techniques like model distillation [LHCG19a] can further bring down the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efficient versions of them for use in appropriate contexts.”
What the sources say
- Not matchedcites LHCG19a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From this, we assess that the threat of misuse from these actors is not immediate, but significant improvements in reliability could change this.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The forums and chat groups monitored are representative of how low- and mid-skill threat actors think and behave.
- Rests on an assumption: APTs do not typically discuss operations in the open.
- Rests on an assumption: Professional threat analysts' assessments about APT activity involving language models are reliable.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conclude that either our conservative method substantially overestimated contamination or that contamination has little effect on performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Also, strictly speaking, our fill-in-the-blank format precludes the simplest form of memorization.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“[RRS20] demonstrate benefits of customizing prediction to entities of interest.”
What the sources say
- Not matchedcites RRS20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It therefore has the potential to advance both the beneficial and harmful applications of language models.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Finally, large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world [BHT + 20].”
What the sources say
- Not matchedcites BHT + 20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Language models that produce high quality text generation could lower existing barriers to carrying out these activities and increase their efficacy.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a different approach is likely to be necessary.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Synthetic tasks such as wordscrambling or defining nonsense words seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although possibly from data that is very different in organization and style than the test data.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright remove problematic results, depending on the severity.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Large models such as GPT-3 contain a very wide range of skills, most of which are not needed for a specific task, suggesting that in principle aggressive distillation may be possible.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%).”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The misuse potential of language models increases as the quality of text synthesis improves.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any socially harmful activity that relies on generating text could be augmented by powerful language models.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Ippolito et al. [IDCBE19] also note that human accuracy at detecting model generated text increases as humans observe more tokens.”
What the sources say
- Not matchedcites [IDCBE19]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to ensure that the sample of participants for each experiment quiz was unique, participants were not allowed to take part in an experiment more than once.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz B.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants were excluded due to internet check fails.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To compare means on the different runs, we performed a two-sample t-test for independent groups for each model against the control.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Model generation selection methods, exclusion criteria, and statistical tests mirror those of the previous experiments.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3 adapts well to many tasks other than the ones explored in the main body of the paper.”
boostedn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The 95% confidence intervals were estimated from the t-distribution of the sample mean.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All data comes from the ground truth datasets in this section, and no samples from GPT-3 are included here.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When plotting a regression line in the graph of average participant accuracy vs model size, we fit a power law of the form ax -b .”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We recruited 718 unique participants to take part in 6 experiments.”
plainn = 718- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“[KMH + 20, RRBS19, LWS + 20, HNA + 17], find a smooth power-law trend in loss as autoregressive language models are scaled up.”
What the sources say
- Not matchedcites KMH + 20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites RRBS19
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites LWS + 20
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each quiz consisted of 25 articles: half (12-13) were human written and half (12-13) were model generated: the articles with human written completions in quiz A had model generated completions in quiz B and vice versa.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The order of quiz question was shuffled for each participant.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“97 participants were excluded for failing an internet check question, leaving a total of 621 participants: 343 male, 271 female, and 7 other.”
plainn = 621- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We first experimented with a few prompts, then generated four samples with no additional editing or selection (sampling at temperature 1 using nucleus sampling [HBFC19] with P = 0 . 9 ).”
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast pre-training at large enough scale appears to offer a 'natural' broad distribution of tasks implicitly contained in predicting the text itself.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
To achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.
“to achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.”
The paper relies on this without support.
Humans do not require large supervised datasets to learn most language tasks; a brief directive or a tiny number of demonstrations is often sufficient to enable a human to perform a new task to at least a reasonable degree of competence.
“humans do not require large supervised datasets to learn most language tasks - a brief directive in natural language (e.g. 'please tell me if this sentence describes something happy or something sad') or at most a tiny number of demonstrations (e.g. 'here are two examples of people acting brave; please give a third example of bravery') is often sufficient to enable a human to perform a new task to”
The paper relies on this without support.
Models have no access to the specific facts that article titles refer to or when the article was written.
“Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written.”
The paper relies on this without support.
The news article titles and subtitles used were not in the models' training data.
“The articles we selected were not in the models' training data and the model outputs were formatted and selected programmatically to prevent human cherry-picking.”
The paper relies on this without support.
CL, A1, and A2 are not bijective.
“Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling.”
The paper relies on this without support.
The forums and chat groups monitored are representative of how low- and mid-skill threat actors think and behave.
“To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed.”
The paper relies on this without support.
APTs do not typically discuss operations in the open.
“Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about possible APT activity involving the use of language models.”
The paper relies on this without support.
Professional threat analysts' assessments about APT activity involving language models are reliable.
“we have consulted with professional threat analysts about possible APT activity involving the use of language models.”
The paper relies on this without support.
The 388 occupations tested are representative of occupations in general.
“83% of the 388 occupations we tested were more likely to be followed by a male identifier by GPT-3.”
The authors state this as an assumption.
The attention operation typically uses less than 10% of the total compute for the models analyzed.
“As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10% of the total compute for the models we are analyzing.”
The paper relies on this without support.
T5 uses an encoder-decoder model, so only half of the parameters are active for each token during a forward or backwards pass.
“Next we note that since T5 uses an encoder-decoder model, only half of the parameters are active for each token during a forward or backwards pass.”
The paper relies on this without support.
The information required to answer DROP questions is in a passage provided to the model.
“The information required to answer the question is in a passage provided to the model, so having seen the passage during training but not the questions and answers does not meaningfully constitute cheating.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Existing work tends to fall on a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019).”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Brown et al., 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“The number of reported contributors increased from 25 for GPT-3 [14] to 284 for GPT-4 [12]”
From The rising costs of training frontier AI models (2024) · cites this paper as [14]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [12]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [12]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs refer to the recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance [9,16].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [9]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, ChatGPT-3 can solve few-shot tasks through in-context learning or sequential reasoning in complex tasks, while GPT-2 or BERT can not do that well.”
From ChatGPT and Deepseek: Can They Predict the Stock Market and Macroeconomy? (2025) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is because the language modeling objective used for many recent large LMs-predicting the next token on a webpage from the internet-is different from the objective 'follow the user's instructions helpfully and safely' (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT17, which excel at information retrieval, nuanced understanding and text generation18,19.”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In recent years, large language models (LLMs) have achieved notable successes, demonstrating significant potential in attaining human-like intelligence [5][6][7][8][9][10].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3 is trained on 175 billion parameters from 45 terabytes of human-created text collected from web pages, Wikipedia, books, and social media data [13].”
From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models process information within a finite attention window, and their internal representations do not persist across interactions, causing earlier details to be forgotten once they fall outside the active context (Brown et al., 2020; Beltagy et al., 2020a).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ICL is constrained by context window length and lacks persistent memory across sessions”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Brown et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Yet, although AI outperforms humans in an ever-broadening range of tasks, from playing poker (20) and Go (21) to translating languages (22) and diagnosing skin cancer (23), it trails far behind when it comes to ToM.”
From Evaluating large language models in theory of mind tasks (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The model itself is large (it has 175 billion parameters), and it was trained on a vast amount of data: hundreds of billions of words from the Internet and books.”
From Using cognitive psychology to understand GPT-3 (2023) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”
From Scaling Instructable Agents Across Many Simulated Worlds (2024) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies demonstrate that pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks [23,150].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3's training on a vast corpus of 45 terabytes of text, sourced from a variety of internet-based entities like Common Crawl and Wikipedia, set a precedent for the scale of data utilised in AI development (Brown et al., 2020;Rudolph et al., 2023a).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent pre-trained LLMs (e.g., GPT-3 [8], ChatGPT [1]) can engage in fuent, multi-turn conversations outof-the-box”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The only adaptable component of LLMs is their in-context learning ability-a (known to be emergent) characteristic of LLMs that enables fast adaption to the context and so perform zero- or few-shot tasks (Brown et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Brown et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs such as GPT-4 (refs. 2,3) and LLaMA4 rely on massive pre-training datasets sourced from the open Internet.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Just as with any other type of AI that is based on large amounts of data, language models can potentially suffer from biases. Most models use training data based on text crawled from the internet. Such text generally has not specifically been checked for bias.”
From Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models (2023) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current mainstream VLMs [209,206,204,208] are usually pre-trained based on large-scale Transformer architectures, which usually have billions or even tens of billions of parameters, and pre-training on large-scale datasets requires a large amount of computational resources and time costs, which limits the training efficiency and scale-up ability of the models, and makes the model deployment face”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [208]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within the past five years, advancements in NLP have achieved huge gains in accuracy when tested against standard datasets [8, 18, 34, 56, 58, 60, 63], with state-ofthe-art accuracy in natural language processing models as high as 99% for certain tasks [8, 34].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Incontext learning facilitates adaptation to novel tasks without parameter updates by leveraging demonstration examples within prompts, with performance significantly influenced by example selection and ordering strategies [369, 103, 1296, 1024, 928, 852, 1148, 352, 582].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [103]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few-shot prompting extends this capability by incorporating limited exemplars to guide model responses, demonstrating task execution through strategic example selection [1371, 405, 103, 552, 794, 1381].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [103]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Modern LLM short-term memory frequently manifests as in-context learning, reflecting models' ability to acquire and process information temporarily within context windows [1189, 103].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [103]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With the unprecedented scale of training and model parameters, the capability of large language models has been dramatically improved, leading to human-like performances in understanding, language synthesizing, common-sense reasoning, etc.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Nowadays, language models are mostly built upon deep neural networks, where two features need to be emphasized. First of all, the majority of language models are based on transformers or their close variations [28, 29].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling up of network and training data lead to the leap-forward of large language models' capability.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“OpenAI has been expanding the model parameters and training data while employing techniques such as RLHF (Reinforcement Learning from Human Feedback) and Instruction Tuning to further fine-tune GPT-3 [3]. This has led to the emergent abilities of models like InstructGPT [37] and subsequent models like ChatGPT”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many works, for example, trace their methodologies for prompted evaluation to Brown et al. (2020), who rather thoroughly document the majority of their evaluation setups, despite not releasing code.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Token-length normalization : each a i 's loglikelihood is divided by m i , its length in tokens, to gain the per-token loglikelihood of each answer. This approach requires no additional LM calls, and is used alternately with raw loglikelihoods for most tasks by Brown et al. (2020).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Brown et al. (2020)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prompt learning methods [2,5,8,16,18,32,42] emerge recently as efficient and effective solutions for adapting pre-trained transformers to the target domain via only training very few parameters (i.e., prompts), and achieve comparable performance with finetuning the whole heavy model.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Brown et al. [5] highlighted that the choice of examples can drastically alter the output of large language models.”
From Demonstration Attack against In-Context Learning for Code Intelligence (2024) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“their offline training paradigm often limits access to customized and up-to-date information.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al., 2020; Lepikhin et al., 2020).”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the LM learns a new task via inference alone by conditioning on a concatenation of the training data as demonstrations, without any gradient updates.”
From Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A limiting factor for in-context learning (ICL) in LLMs is the context window, restricting prior research to the few-shot ICL regime.”
From Many-Shot In-Context Learning (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Brown et al. [8] reported improved performance as you increase the number of examples (up to 64) for in-context learning in LLMs , and later works corroborated this finding [39].”
From Many-Shot In-Context Learning (2024) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The abilities of models, which are not publicly available, for example LaMDA [8], DALL-E [97], GPT-3 [3], ChatGPT and GPT-4 [39], are virtually impossible to verify.”
From Deanthropomorphising NLP: Can a language model be conscious? (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ACE offers a flexible and efficient alternative to conventional model finetuning, as adapting contexts is generally cheaper than updating model weights”
From Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (2025) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The superiority of ChatGPT compared to its GPT-based predecessors can be linked to its ability to respond to multiple languages generating refined and highly sophisticated responses based on advanced modeling [6,7].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Developed by OpenAI (OpenAI, L.L.C., San Francisco, CA, USA), ChatGPT etymology is related to being a chatbot (a program able to understand and generate responses using a text-based interface) and is based on the generative pre-trained transformer (GPT) architecture [6,7].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompting methods are particularly useful in the latter case [9,32,117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger [9,88,139].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The increasingly large context length of LLMs enables in-context learning (ICL) [Brown et al., 2020] by incorporating more documents, enhancing the model's ability to capture intricate writing patterns.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI models such as GPT-3 [120] and DALL·E https://openai.com/index/dall-e-3/, remain reactive generating content solely in response to prompts without maintaining a persistent state or engaging in iterative reasoning.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [120]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These networks are trained through unsupervised machine learning methods on enormous amounts of text from various sources, predominantly open-domain web resources, with academic content constituting only a small fraction (Brown et al., 2020).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, many benchmarks are susceptible to contamination [7, 13, 15, 20], as exemplified by recent works [5, 7, 15] and social media posts [1, 19]. This risk arises because widely used open-source repositories-particularly those distributed under permissive licenses (e.g., MIT, Apache 2.0, BSD)-are prime candidates for inclusion in the large-scale web-crawled corpora used to pre-train LLMs [3].”
From SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Figure 1 shows that prompt tuning beats GPT-3 prompt design by a large margin, with prompttuned T5-Small matching GPT-3 XL (over 16 times larger), and prompt-tuned T5-Large beating GPT-3 175B (over 220 times larger).”
From The Power of Scale for Parameter-Efficient Prompt Tuning (2021) · cites this paper as Brown et al. (2020)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.”
From LoRA: Low-Rank Adaptation of Large Language Models (2021) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).”
From LoRA: Low-Rank Adaptation of Large Language Models (2021) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even pretrained LLMs cannot solve these tasks robustly (Brown et al., 2020;Chowdhery et al., 2022;Anil et al., 2022) without careful prompting (Zhou et al., 2022b).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Agentic evaluations are inherently stochastic processes.”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Brown et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The underlying LLM models sample from probability distributions rather than deterministically computing outputs (Brown et al. 2020; Holtzman et al. 2020; He and Lab 2025), introducing trial-to-trial variance..”
From Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (2025) · cites this paper as Brown et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2022) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“using temporal splits (or splits beyond random ones) is still the exception rather than the rule, as evidenced by many contemporary LM (Brown et al., 2020; Gao et al., 2021) and downstream tasks (Lewis et al., 2020a) that are affected by test data contamination.”
From Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods.”
From Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“using the demonstrations with gold labels significantly improves the performance over no demonstrations”
From Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as [Brown et al. 2020]
The cited source says what the claim attributes to it.
In the citing paper
“These models have shown exceptional performance in text generation-related tasks, and significant few-shot learning capabilities.”
From A review of LLMs and their applications in the architecture, engineering and construction industry (2025) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“training our 175B SFT model requires 4.9 petaflops/s-days and training our 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for GPT-3 (Brown et al., 2020).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”
From Grounded Copilot: How Programmers Interact with Code-Generating Models (2023) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The LLM used in this work (i.e., GPT-3 [13]) is open to public use with easy registration and no user education.”
From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“However, as previously mentioned, the use of these models should be done with caution, as they also have limitations such as lack of interpretability and potential for bias, unexpected brittleness in relatively simple tasks [9] which need to be addressed.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks (Brown et al., 2020; Achiam et al., 2023; Wei et al., 2022), yet they remain limited in their ability to maintain and reason over long-term context.”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“GPT-3 is an autoregressive language model (2), which utilizes the transformer architecture (9)—a deep learning model that heavily relies on the mechanism of self-attention—to produce human-like text.”
From Using cognitive psychology to understand GPT-3 (2023) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“In this experiment, we evaluate in the few-shot regime using a 137B parameter pre-trained decoder-only model, as previous work indicates that very large models may be able to perform additions and multiplications with 3 or fewer digits few-shot (Brown et al., 2020).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“For example, GPT-3 struggles to perform few-shot addition on numbers with greater than three digits (Brown et al., 2020).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”
From Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“From simple next-token predictors [1, 2], large language models (LLMs) have evolved to exhibit graduate-level STEM proficiency [3-5], generate hypotheses [6, 7], synthesize literature [8], and draft manuscripts [9].”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Min et al. [44] has shown that rather than learning new tasks, in-context learning [42] works by activating patterns from the training data.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“Using in-context learning [42], these methods generate reasonable plans in structured language, but requires pre-defined action primitives to execute.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“However, pretrained language models, especially large ones, appear to have significantly improved sample efficiency when learning new tasks [1,59,60]”
From Symbols and grounding in large language models (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Early work on few-shot learning (Brown et al., 2020) and prompt engineering (Wei et al., 2022b; Kojima et al., 2022; Xu et al., 2023c) found that the LLM could be further biased towards high-quality productions by pre-processing the input string.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“GPT-3 (Brown et al., 2020) unlocked flexible few-shot and zero-shot reasoning for NLP tasks”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“LLM workloads contain massive prefix redundancy: system prompts repeated across thousands of requests, few-shot examples [32], retrieval-augmented contexts.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“in a different but plausible world in which MMLU-style prompting was the standard introduced in Brown et al. (2020), the perceived performance of GPT-NeoX-20B and other models trained on the Pile (Gao et al., 2020) would be extremely low, despite strong results that would be observed in the 'cloze' setting.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Brown et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Through crowdworkers, students, and GPT-3 [58], we also associate each object with our fully simulatable object states: for example, apple is associated with cooked and sliced , but not toggledOn .”
From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“All properties that apply to many of the 2,964 leaf synsets and therefore require a large-scale annotation are done by either GPT-3 [58] or crowdworkers.”
From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“LLMs consist of neural networks trained on extensive text corpora to identify probable word and language patterns, which are then stored as weights within the neural network (Brown et al., 2020; Vaswani et al., 2017).”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLMs) are designed to predict probable language outputs in response to specific prompts, such as predicting the most likely word following a sequence of words or generating sentences or paragraphs as plausible responses to a given textual prompt.”
From Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Initial audits also highlight that language models reproduce stereotypes and biases [23, 54, 83]”
From Co-Writing with Opinionated Language Models Affects Users’ Views (2023) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“ChatGPT is a variant of the GPT-3 (Generative Pre-trained Transformer 3, Brown et al., 2020) artificial intelligence language model developed by OpenAI.”
From Chatting and cheating: Ensuring academic integrity in the era of ChatGPT (2024) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“In-context learning (ICL) is a key ability that enables LLMs to adapt to new tasks on the fly (Brown et al., 2020).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The original GPT-3 paper demonstrated that examples substantially improved the performance of GPT-3 on a battery of tasks such as question answering and language translation [9].”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, large language models (LLMs) (Brown et al., 2020;Ouyang et al., 2022;OpenAI, 2023) have demonstrated exceptional language generation capabilities across various domains.”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“These practical applications of language technology are increasingly reliant on approaches based on trained Large Language Models (LLMs) [23, 41, 104, 111], whereby models are first exposed to as large and varied a collection of language data as possible with the aim of extracting 'general' properties of a language of interest.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“they can recall and use this information more accurately with increasing scale (Brown et al., 2020).”
From Memory Layers at Scale (2024) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Brown et al. (2020) report that the GPT-3 model with 175 billion parameters used 3 . 14 · 10 23 floating point operations (FPOs) of compute to train using NVIDIA V100 GPUs on a cluster provided by Microsoft.”
From Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models (2020) · cites this paper as Brown et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“These models are designed to provide a strong foundation of general knowledge and skills that can be built upon and adapted to new situations via fine-tuning or prompting with demonstrations (Brown et al., 2020).”
From Human-Timescale Adaptation in an Open-Ended Task Space (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Foundation models typically require large, diverse datasets to achieve their generality (Brown et al., 2020; Mahajan et al., 2018; Schuhmann et al., 2022; Sun et al., 2017; Zhai et al., 2022).”
From Human-Timescale Adaptation in an Open-Ended Task Space (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Few-shot prompting (also referred to as”
From Self-Refine: Iterative Refinement with Self-Feedback (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“We utilize API calls to a publicly available LLM (Brown et al., 2020) which, whilst sufficient in the context of our investigation, exposes us to variable inference time calls and unforeseen model updates.”
From Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”
From Teaching Large Language Models to Self-Debug (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Because the capabilities of these models significantly improve with model size (Brown et al., 2020; Radford et al., 2019; Tamkin et al., 2021)”
From Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Developers of models like GPT-3 seem to be aware of this problem and discuss gender, race, and religious biases in their article (Brown et al., 2020).”
From Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“LLMs demonstrate proficiency in generating text [11, 14, 24] by producing fluent and precise linguistic expressions.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Our language module uses BioGPT23, an autoregressive generative model based on the Transformer24 decoder architecture of GPT-325, trained on 15 million biomedical articles from PubMed with a vocabulary size of 42,384.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“State-of-the-art LLMs [66, 67, 7, 52] are generally implemented using the Transformer architecture, which is a multi-layer feedforward network equipped with attention mechanism [71].”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLMs) have demonstrated the capability to store extensive knowledge during pre-training and recall it during inference”
From AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (2024) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“There has been tremendous progress in language models in recent years, where the emergence of large language models, represented by GPT-3 [3], marks an important milestone for the entire AI community.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“In-context learning [167-173] is one of the emergent abilities of LLMs that differentiate LLMsfrompreviouspre-trainedlanguagemodels,where,givenanaturallanguageinstruction and task demonstrations, LLMs would generate the output by completing the word sequence without training or tuning [3].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“In-context learning [167-173] is one of the emergent abilities of LLMs that differentiate LLMsfrompreviouspre-trainedlanguagemodels,where,givenanaturallanguageinstruction and task demonstrations, LLMs would generate the output by completing the word sequence without training or tuning [3].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models are widely known to leverage extensive amounts of open-domain knowledge during their training and fine-tuning processes. These knowledge sources include well-known references such as Wikipedia, books, and various websites [3].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“Although this approach is nonstandard, it is provided in lm-eval under the option acc mutual info , and used selectively by Brown et al. (2020) and Askell et al. (2021) for certain tasks.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Brown et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Provided with these three primitive operations, we are able to implement the major ways in the literature that have been used to evaluate LMs (Gao et al. (2020), Brown et al. (2020), inter alia ).”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Brown et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“As these models are trained on human-written texts, the content generated by these tools can be quite difficult to distinguish from human-written content.”
From Testing of detection tools for AI-generated text (2023) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Brown et al. [11] first show that GPT-3 has the ability to learn from a few examples in the context without parameter update.”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“While we relied on a zero-shot strategy (see Brown et al., 2020)”
From ChatGPT for President! Presupposed content in politicians versus GPT-generated texts (2025) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“This contrasts with other NLP tasks, where performance improves with model size.”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Although state-of-the-art models, such as GPT-3, have been introduced for over two years (Brown et al., 2020), they have yet to be widely applied to automate educational tasks.”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100 × : from only a few thousand tokens in GPT-3 [8] and Llama 2 [57] to 1M tokens in Gemini 1.5 Pro [16].”
From Many-Shot In-Context Learning (2024) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“This means that our conclusions, especially regarding consciousness, apply to other models such as BERT [1], GPT-3 [2], T5 [4] and future models using the same, or similar, architectures.”
From Deanthropomorphising NLP: Can a language model be conscious? (2024) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Behaving in an aligned fashion is just another capability, and many works have shown that larger models are more capable [Kaplan et al., 2020, Rosenfeld et al., 2019, Brown et al., 2020], finetune with greater sample efficiency [Henighan et al., 2020, Askell et al., 2021], and do not suffer significantly from forgetting [Ramasesh et al., 2022].”
From Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“First, the large-scale pretraining of LLMs equips them with vast knowledge across a wide range of domains (Brown et al. 2020, Touvron et al. 2023).”
From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The GPT architecture utilizes a neural network to process natural language, thus generating responses based on the context of input text [7].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“In-context learning (McCann et al. 2018; Radford et al. 2018; Brown et al. 2020) is a key mechanism for foundation model programming.”
From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”
From Training Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice (2026) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“These few-shot demonstrations take advantage of the ability of strong language models to learn repetitive patterns [9].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Qwen3 models utilize Qwen's tokenizer (Bai et al., 2023), which implements byte-level byte-pair encoding (BBPE, Brown et al., 2020;Wang et al., 2020;Sennrich et al., 2016) with a vocabulary size of 151,669.”
From Qwen3 Technical Report (2025) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“LLMs have shown promising performances in various classification tasks (Brown et al., 2020), and prior work (Don-Yehiya et al., 2024; Shaikh et al., 2025) has also explored prompting LLMs (specifically GPT-4o-mini) to classify user feedback in multi-turn user-LLM interactions.”
From User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“In-context learning (Brown et al., 2020) allows the model to learn from input examples, enabling it to understand and manage each new task.”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“We note that increasing the number of parameters improved model performance, consistent with recent advances in NLP7.”
From Effective gene expression prediction from sequence by integrating long-range interactions (2021) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing (NLP)6,7 and were also recently applied to model short DNA sequences8.”
From Effective gene expression prediction from sequence by integrating long-range interactions (2021) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin.”
From The Power of Scale for Parameter-Efficient Prompt Tuning (2021) · cites this paper as Brown et al. (2020)
The cited source says what the claim attributes to it.
In the citing paper
“Following Kaplan et al. (2020) and the training setup of GPT-3 (Brown et al., 2020), many of the recently trained large models have been trained for approximately 300 billion tokens (Table 1), in line with the approach of predominantly increasing model size when increasing compute.”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Pre-training constitutes the foundational stage for LLMs, predominantly utilizing a transformer-based architecture following the paradigm established by GPT [29, 251, 252], and further developed by OPT[372], Falcon [243], and Llama-2 [300].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“A significant part of the success of LLMS is their ability to learn new tasks by being given even a few examples as context in their prompts [97].”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [97]
The cited source says what the claim attributes to it.
In the citing paper
“Our hypothesis that episodic memories could be recalled and used in ways which lead to unpredictable and potentially undesirable agent behavior has a strong parallel in existing work on the effects of examples given to large language models.”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [97]
The cited source says what the claim attributes to it.
In the citing paper
“AI agents are able to both search the internet [119], [120] and learn new skills in this manner today [97], [121].”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [97]
The cited source says what the claim attributes to it.
In the citing paper
“At the core of these systems are large language models (LLMs), which generate outputs by predicting likely token sequences based on patterns learned from large-scale training data (Brown et al. 2020;Radford et al. 2018;Vaswani et al. 2017).”
From Machine marketing: rethinking the customer in the age of generative AI (2026) · cites this paper as Brown et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The official estimate indicates that GPT-3 consumes an order of 0.4kWh of electricity to generate 100 pages of content, equivalent to roughly 0.004kWh per page.4”
From Making AI Less 'Thirsty' (2025) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al., 2020), they often need training to achieve the best results.”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2023) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“As a few examples, GPT-3 175B achieved new state of the art on the TriviaQA and PiQA question-answering benchmarks (Brown et al., 2020); PaLM 540B achieved new state of the art on three arithmetic reasoning benchmarks (Chowdhery et al., 2022); and the multimodal Flamingo 80B model achieved new state of the art on six visual question answering benchmarks (Alayrac et al., 2022).”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Recently, increasing model size has led to substantial improvements in perplexity, downstream tasks, and few-shot learning ability (Kaplan et al., 2020; Brown et al., 2020).”
From Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021) · cites this paper as Brown et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“In comparison, its predecessor GPT-3 models come in different sizes and contain from 125 million parameters for the smallest model to 175 billion for the largest [10].”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [10]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Knowledge of a robot body is often known, hence reaching a point in the 3D space is feasible using motion planning [53, 59, 60].”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [60]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“GPT-3 is used for all properties (total of eight) for which the Hamming distance and false-positive rate for a sample compared to a human-annotated ground-truth are both less than 10%.”
From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [58]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“empirical observations indicate NLP modeling (including Transformer) does not require a deep structure (Vaswani et al., 2017; Brown et al., 2020)”
From Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice (2022) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“MoE-LLaVA [208] uses a Mixture of Experts (MoE) architecture, activating only the relevant expert networks to reduce computational burden and improve reasoning efficiency, especially on large datasets.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [208]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“All HistoGPT models consistently produced text with high cosine similarity to the ground truth, according to the sentence embeddings provided by BioBERT33 and GPT-3-ADA25 (Fig. 3d).”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“These components are fundamental for task decomposition, logical inference, action execution, and iterative improvement, following established patterns in agentic systems literature.”
From Understanding and Optimizing Agentic Workflows via Shapley value (2025) · cites this paper as Brown et al. 2020
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“It worth noting that Codex is more powerful than CodeGPT since it is a large-scale pretrained model that is trained on all source codes in GitHub based on GPT-3 (Brown et al., 2020).”
From ReACC: A Retrieval-Augmented Code Completion Framework (2022) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Virtually all existing architectures for dense prediction are based on convolutional networks [6,31,34,42,49,50,53].”
From Vision Transformers for Dense Prediction (2021) · cites this paper as [42]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Brown et al., 2020
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing work had to restrict their models to a small list of selected genes.”
From Large-scale foundation model on single-cell transcriptomics (2024) · cites this paper as [10]
Experimental — accuracy not yet measuredIn the citing paper
“Models also seem to (often) be able to use novel words appropriately [69, 73].”
From Dissociating language and thought in large language models (2024) · cites this paper as [69]
In the citing paper
“GPT-3 performs well on two-digit addition and subtraction but not on more complex tasks, such as three-digit addition or two-digit multiplication [69].”
From Dissociating language and thought in large language models (2024) · cites this paper as [69]
In the citing paper
“Here, scaling GPT-3 to around 3 · 10 23 training FLOPs (175B parameters) failed to unlock above-random one-shot prompting performance.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Brown et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 3.9.2 Word Scrambling and Manipulation Tasks · p. 23
“the full GPT-3 model achieving 66.9% on removing random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task”
§ 3.9.1 Arithmetic · p. 23
“| GPT-3 Few-shot | 37.9 | 15.1 | 39.7 | 67.2 | 0.44 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.9.1 Arithmetic · p. 22
“80.2% at 3 digit addition”
§ 3.9.1 Arithmetic · p. 23
“| GPT-3 Few-shot | 100.0 | 98.9 | 80.4 | 94.2 | 25.5 | 26.8 | 9.3 | 9.9 | 29.2 | 21.3 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| GPT-3 XL | 1.3B | 24 | 2048 | 24 | 128 | 1M | 2 . 0 × 10 - 4 |”
Reported 2048 · recomputed 3072
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| GPT-3 Large | 81 | 24 | 46:28:2 | 37 | 216:200 |”
Reported 81 · recomputed 100
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.5 Common Sense Reasoning · p. 17
“GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot”
§ 3.4 Winograd-Style Tasks · p. 17
“| GPT-3 Zero-Shot | 80.5 * | 68.8 | 51.4 | 57.6 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Self-citationLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Cited by
Sorted by citations
Items 2,801–2,900 of 33,402
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., & Ke, G. (2022) · doi:10.26434/chemrxiv-2022-jjm0j · Cited by 81
Vincent, N., Li, H., Tilly, N., Chancellor, S., & Hecht, B. J. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445885 · Cited by 81
Compressing Visual-linguistic Model via Knowledge Distillation
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00146 · Cited by 81
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
Xu, H., Durme, B. V., & Murray, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.534 · Cited by 81
Semantic Structure in Deep Learning
Pavlick, E. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 82
True Few-Shot Learning with Prompts—A Real-World Perspective
Schick, T., & Schütze, H. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00485 · Cited by 81
A Memristors‐Based Dendritic Neuron for High‐Efficiency Spatial‐Temporal Information Processing
Li, X., Zhong, Y., Chen, H., Tang, J., Zheng, X., Sun, W. B., Li, Y., Wu, D. M., Gao, B., Hu, X., et al. (2022) · Advanced Materials · doi:10.1002/adma.202203684 · Cited by 81
Hernandez, I., & Nie, W. (2022) · Personnel Psychology · doi:10.1111/peps.12543 · Cited by 83
YOLO-Former: Marrying YOLO and Transformer for Foreign Object Detection
Dai, Y., Liu, W., Wang, H., Xie, W., & Long, K. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3219468 · Cited by 81
With Shared Microexponents, A Little Shifting Goes a Long Way
Rouhani, B. D., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G. V., et al. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589351 · Cited by 81
Single-shot optical neural network
Bernstein, L., Sludds, A., Panuski, C., Trajtenberg‐Mills, S., Hamerly, R., & Englund, D. R. (2023) · Science Advances · doi:10.1126/sciadv.adg7904 · Cited by 81
Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification
Wu, J., Chang, D., Sain, A., Li, X., Ma, Z., Cao, J., Guo, J. H., & Song, Y.-Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25383 · Cited by 81
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
Dehallucinating Large Language Models Using Formal Methods Guided Iterative Prompting
Jha, S., Jha, S. K., Lincoln, P. D., Bastian, N. D., Velasquez, A., & Neema, S. K. (2023) · IEEE International Conference on Assured Autonomy (ICAA) · doi:10.1109/icaa58325.2023.00029 · Cited by 81
Integrating action knowledge and LLMs for task planning and situation handling in open worlds
Ding, Y. F., Zhang, X., Amiri, S., Cao, N., Yang, H., Kaminski, A., Esselink, C., & Zhang, S. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10133-5 · Cited by 81
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 83
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Fei, J., Wang, T., Zhang, J., He, Z., Wang, C., & Zheng, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00291 · Cited by 81
Huang, G., Li, Y., Jameel, S., Long, Y., & Papanastasiou, G. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.05.004 · Cited by 81
Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling
Park, M., Kim, S.-J., Lee, S., Kwon, S., & Kim, K. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651122 · Cited by 81
Irfan, B., Kuoppamäki, S., & Skantze, G. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1363713 · Cited by 82
Bhayana, R., Nanda, B., Dehkharghanian, T., Deng, Y., Bhambra, N., Elias, G. J. B., Datta, D., Kambadakone, A. R., Shwaartz, C. G., Moulton, C.-A. E., et al. (2024) · Radiology · doi:10.1148/radiol.233117 · Cited by 81
SeD: Semantic-Aware Discriminator for Image Super-Resolution
Li, B., Li, X., Zhu, H., Jin, Y., Feng, R., Zhang, Z., & Chen, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02436 · Cited by 81
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 84
Generative Artificial Intelligence in Qualitative Data Analysis: Analyzing—Or Just Chatting?
Nguyen, D. C., & Welch, C. (2025) · Organizational Research Methods · doi:10.1177/10944281251377154 · Cited by 81
LLM as a System Service on Mobile DevicesPreprint
Yin, W., Xu, M., Li, Y., & Liu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.11805 · Cited by 82
SafeArena: Evaluating the Safety of Autonomous Web AgentsPreprint
Tur, A. D., Meade, N., Lù, X. H., Zambrano, A., Patel, A., Durmus, E., Gella, S., Stańczak, K., & Reddy, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.04957 · Cited by 82
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80
Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
Ullman, T. (2023) · arXiv · doi:10.48550/arxiv.2302.08399 · Cited by 81
AI Assistance in Legal Analysis: An Empirical Study
Choi, J. H., & Schwarcz, D. (2023) · doi:10.2139/ssrn.4539836 · Cited by 80
Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00345-9 · Cited by 80
AI Companions Reduce Loneliness
De Freitas, J., Uğuralp, A. K., Uğuralp, Z., & Puntoni, S. (2024) · doi:10.2139/ssrn.4893097 · Cited by 80
ProsocialDialog: A prosocial backbone for conversational agents
Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., Choi, Y., & Sap, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.267 · Cited by 80
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80
Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification
Wang, Z., & Ji, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20472 · Cited by 81
Dickinson, Q., & Meyer, J. G. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009736 · Cited by 80
On cross-lingual retrieval with multilingual text encoders
Litschko, R., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2022) · Information Retrieval · doi:10.1007/s10791-022-09406-x · Cited by 80
Mathematical Aspects of Deep Learning
Berner, J., Grohs, P., Gitta Kutyniok, B., & Petersen, P. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009025096 · Cited by 80
Unlocking Practical Applications in Legal Domain
Šavelka, J. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595161 · Cited by 82
Scalable Educational Question Generation with Pre-trained Language Models
Bulathwela, S., Muse, H., & Yılmaz, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_27 · Cited by 80
Lanzi, P. L., & Loiacono, D. (2023) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3583131.3590351 · Cited by 80
Corrupted by Algorithms? How AI-generated and Human-written Advice Shape (Dis)honesty
Leib, M., Köbis, N., Rilke, R. M., Hagens, M., & Irlenbusch, B. (2023) · The Economic Journal · doi:10.1093/ej/uead056 · Cited by 80
Current cases of AI misalignment and their implications for future risks
Dung, L. (2023) · Synthese · doi:10.1007/s11229-023-04367-0 · Cited by 80
Wang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 80
Finding Support Examples for In-Context Learning
Li, X., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.411 · Cited by 81
Beyond Traditional Assessment: Exploring the Impact of Large Language Models on Grading Practices
Fagbohun, O., Iduwe, N. P., Abdullahi, M., Ifaturoti, A., & Nwanna, M. (2024) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/19 · Cited by 80
Hajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 87
A Historical Survey of Advances in Transformer Architectures
Sajun, A. R., Zualkernan, I. A., & Sankalpa, D. (2024) · Applied Sciences · doi:10.3390/app14104316 · Cited by 80
A Survey on the Use of Large Language Models (LLMs) in Fake News
Papageorgiou, E., Chronis, C., Varlamis, I., & Himeur, Y. (2024) · Future Internet · doi:10.3390/fi16080298 · Cited by 80
Evolving code with a large language model
Hemberg, E., Moskal, S., & O’Reilly, U.-M. (2024) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-024-09494-2 · Cited by 80
Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives
Liu, Y., Yue, J., Xia, S., Ghamisi, P., Xie, W., & Fang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3464685 · Cited by 81
ST-LLM: Large Language Models Are Effective Temporal Learners
Liu, R., Li, C., Tang, H., Ge, Y., Shan, Y., & Li, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_1 · Cited by 82
Addressing the antibody germline bias and its effect on language models for improved antibody design
Olsen, T. H., Moal, I. H., & Deane, C. M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae618 · Cited by 80
Automating construction contract review using knowledge graph-enhanced large language models
Zheng, C., Wong, S., Su, X., Tang, Y., Nawaz, A., & Kassem, M. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106179 · Cited by 83
Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection
Liu, J., Zhang, C., Qian, J., Ma, M., Qin, S., Bansal, C., Lin, Q., Rajmohan, S., & Zhang, D. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737239 · Cited by 80
Extending Machine Language Models toward Human-Level Language UnderstandingPreprint
McClelland, J. L., Hill, F., Rudolph, M., Baldridge, J., & Schütze, H. (2019) · arXiv (Cornell University) · doi:10.5282/ubm/epub.72201 · Cited by 81
Counterfactual Generative NetworksPreprint
Sauer, A., & Geiger, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06046 · Cited by 80
Epistemic Neural NetworksPreprint
Osband, I., Wen, Z., Asghari, S. M., Dwaracherla, V., Ibrahimi, M., Lu, X., & Roy, B. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08924 · Cited by 80
Step-unrolled Denoising Autoencoders for Text GenerationPreprint
Savinov, N., Chung, J., Bińkowski, M., Elsen, E., & van den Oord, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.06749 · Cited by 83
Huang, K., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2023) · PsyArXiv (OSF Preprints) · doi:10.1016/j.jml.2024.104510 · Cited by 80
CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingPreprint
Chiu, Y. Y., Jiang, L., Lin, B. Y., Park, C. Y., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetkov, Y., Shwartz, V., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.02677 · Cited by 80
Orthogonal Subspace Learning for Language Model Continual Learning
Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.715 · Cited by 79
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79
“GPT-3 175B (Brown et al., 2020) served as the teacher model for decomposing complex problems into a series of simpler substeps (we report the prompts used in Appendix Table 6).”
BackgroundQuantum Neural Network for Quantum Neural Computing
Zhou, M.-G., Liu, Z.-P., Yin, H.-L., Li, C.-L., Xu, T.-K., & Chen, Z.-B. (2023) · Research · doi:10.34133/research.0134 · Cited by 79
Compressing Context to Enhance Inference Efficiency of Large Language Models
Li, Y., Dong, B., Guerin, F., & Lin, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.391 · Cited by 253
Language Models are Open Knowledge Graphs
Wang, C., Liu, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2010.11967 · Cited by 79
From Twitter to traffic predictor: Next-day morning traffic prediction using social media data
Yao, W., & Qian, S. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102938 · Cited by 79
BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems
Asyrofi, M. H., Yang, Z., Yusuf, I. N. B., Kang, H. J., Thung, F., & Lo, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3136169 · Cited by 81
Finetuning Pretrained Transformers into RNNs
Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.830 · Cited by 79
DEMix Layers: Disentangling Domains for Modular Language Modeling
Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., & Zettlemoyer, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.407 · Cited by 81
Powers, C. J., Devaraj, A., Ashqeen, K., Dontula, A., Joshi, A., Shenoy, J., & Murthy, D. (2023) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2023.100164 · Cited by 79
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
Fan, W.-C., Chen, Y., Chen, D., Yu, C., Yuan, L., & Wang, Y.-C. F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25133 · Cited by 80
ConvGQR: Generative Query Reformulation for Conversational Search
Mo, F., Mao, K., Zhu, Y., Wu, Y., Huang, K., & Nie, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.274 · Cited by 79
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Kazemi, M., Kim, N., Bhatia, D. S., Xu, X., & Ramachandran, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.361 · Cited by 79
Parameter-Efficient Fine-Tuning without Introducing New Latency
Liao, B., Meng, Y., & Monz, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.233 · Cited by 79
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., & Li, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.765 · Cited by 79
A Systematic Survey of Chemical Pre-trained Models
Xia, J., Zhu, Y., Du, Y., & Li, S. Z. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/760 · Cited by 79
Multiview Compressive Coding for 3D Reconstruction
Wu, C.-Y., Johnson, J., Malik, J., Feichtenhofer, C., & Gkioxari, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00875 · Cited by 79
Rationalization for explainable NLP: a survey
Gurrapu, S., Kulkarni, A., Huang, L., Lourentzou, I., & Batarseh, F. A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1225093 · Cited by 79
Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Lee, D., Ahrabian, K., Jin, W., Morstatter, F., & Pujara, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.36 · Cited by 80
Diffusion Models as Masked Autoencoders
Wei, C., Mangalam, K., Huang, P.-Y., Li, Y., Fan, H., Hu, X., Wang, H., Xie, C., Yuille, A., & Feichtenhofer, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01492 · Cited by 80
He, Y., Zhou, X., Chang, C. H., Chen, G., Liu, W., Li, G., Fan, X., Sun, M., Miao, C., Huang, Q., et al. (2024) · Molecular Cell · doi:10.1016/j.molcel.2024.01.021 · Cited by 79
Transformer-Based Named Entity Recognition in Construction Supply Chain Risk Management in Australia
Shishehgarkhaneh, M. B., Moehler, R., Fang, Y., Hijazi, A. A., & Aboutorab, H. (2024) · IEEE Access · doi:10.1109/access.2024.3377232 · Cited by 79
Connecting Speech Encoder and Large Language Model for ASR
Yu, W., Tang, C., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445874 · Cited by 79
A critical review of large language models: Sensitivity, bias, and the path toward specialized AI
Hajikhani, A., & Cole, C. H. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00310 · Cited by 79
Guo, Y., Ovadje, A., Al-Garadi, M. A., & Sarker, A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae210 · Cited by 79
Analysis of LLMs for educational question classification and generation
Faraby, S. A., Romadhony, A., & Adiwijaya, A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100298 · Cited by 79
ChatEV: Predicting electric vehicle charging demand as natural language processing
Qu, H., Li, H., You, L., Zhu, R., Yan, J., Santi, P., Ratti, C., & Yuen, C. (2024) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2024.104470 · Cited by 79
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
Shaking the foundations: delusions in sequence models for interaction and controlPreprint
Ortega, P. A., Kunesch, M., Delétang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot, B., Pérolat, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10819 · Cited by 79
Robust and Efficient Medical Imaging with Self-SupervisionPreprint
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., MacWilliams, P., Mahdavi, S. S., Wulczyn, E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09723 · Cited by 79
Do LLM Agents Exhibit Social Behavior?Preprint
Leng, Y., & Yuanyuan (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2312.15198 · Cited by 79
Sentiment spin: Attacking financial sentiment with GPT-3
Leippold, M. (2023) · Finance Research Letters · doi:10.1016/j.frl.2023.103957 · Cited by 78
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
Prasad, A., Hase, P., Zhou, X., & Bansal, M. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.277 · Cited by 78
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78
Heinzerling, B., & Inui, K. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.153 · Cited by 78
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78
Metadata
- Authors
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 33
- Pages
- 1877-1901
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.14165
- OpenAlex
- W3030163527
- Semantic Scholar
- 90abbc2cf38462b954ae1b772fac9532e2ccd8b0
- Cited by
- 23,372
- References
- 127
- Citations by year
- 2026: 403 · 2025: 4739 · 2024: 7286 · 2023: 6380 · 2022: 2283 · 2021: 1844 · 2020: 427
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Funders
- National Science Foundation (2112606)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- few-shot learning, GPT-3, large language models, task-agnostic learning, pre-training, fine-tuning, autoregressive language models, domain adaptation, text-to-text translation, question answering, cloze task, text generation, societal impact of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
