Language Models are Few-Shot Learners

In the paper

“On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting.”

In the paper

“On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting.”

In the paper

“GPT-3 has a noticeable skew in its performance depending on language direction. For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction.”

In the paper

“GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA's test server).”

In the paper

“On the 'Easy' version of the dataset (questions which either of the mentioned baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a fine-tuned RoBERTa baseline from [KKS + 20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites KKS + 20

In the paper

“Although we find that data contamination has a minimal effect on GPT-3's performance on most datasets, we do identify a few datasets where it could be inflating results, and we either do not report results on these datasets or we note them with an asterisk, depending on the severity.”

In the paper

“However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3's few-shot results by 27% on the challenge set and 22% on the easy set.”

In the paper

“We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.”

In the paper

“showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance.”

In the paper

“This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [LDL19] but improves over previous zero-shot results by roughly 10%.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LDL19

In the paper

“At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.”

In the paper

“On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM.”

In the paper

“Overall, on one of the three datasets GPT-3's one-shot matches the open-domain fine-tuning SOTA. On the other two datasets it approaches the performance of the closed-book SOTA despite not using fine-tuning.”

In the paper

“Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a significant improvement is observed on OpenBookQA.”

In the paper

“When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase of over 18% from the previous state-of-the-art.”

In the paper

“On DROP [DWD + 19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL + 19].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites DWD + 19
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites RLL + 19

In the paper

“outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [ZHR + 19] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites ZHR + 19

In the paper

“Model performance improves with the addition of a natural language task description, and with the number of examples in the model's context, K .”

In the paper

“For both Fr-En and De-En, few shot GPT-3 outperforms the best supervised result we could find”

In the paper

“On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting.”

In the paper

“Although GPT-3's training data is still primarily English (93% by word count), it also includes 7% of text in other languages.”

In the paper

“Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50.”

In the paper

“Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion parameter model.”

In the paper

“PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance”

In the paper

“Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held by fine-tuned models).”

In the paper

“GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.”

In the paper

“Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to fine-tuned models operating in the same closed-book setting.”

In the paper

“Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with K = 70 ).”

In the paper

“GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting”

In the paper

“GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work.”

In the paper

“On COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where first place is held by a fine-tuned 11 billion parameter model (T5).”

In the paper

“On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings”

In the paper

“On the 'Challenge' version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot setting, and 51.5% in the few-shot setting.”

In the paper

“However, providing only a single example demonstration for each translation task improves performance by over 7 BLEU and nears competitive performance with prior work.”

In the paper

“The one-shot result improves by 3.7% and matches the SOTA for an open-domain QA system which not only fine-tunes but also makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [LPP + 20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LPP + 20

In the paper

“GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings.”

In the paper

“On OpenBookQA [MCKS18], GPT-3 improves significantly from zero to few shot settings but is still over 20 points short of the overall SOTA.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites MCKS18

In the paper

“Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.”

In the paper

“On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites RJL18

In the paper

“Larger values of K are usually but not always better, so when a separate development and test set are available, we experiment with a few values of K on the development set and then run the best value on the test set.”

In the paper

“GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI + 18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites RCM19
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites CHI + 18

In the paper

“For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting.”

In the paper

“On RACE [LXL + 17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LXL + 17

In the paper

“The average occupation bias - measured as 1 n jobs ∑ jobs log( P (female | Context) P (male | Context)) ) - was -1 . 11 for the Neutral Variant, -2 . 14 for the Competent Variant and -1 . 15 for the Incompetent Variant.”

In the paper

“There is a significant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations.”

In the paper

“as well as on a subset of reading comprehension tasks.”

In the paper

“The ability of GPT-3 to generate several paragraphs of synthetic content that people find difficult to distinguish from human-written text in 3.9.4 represents a concerning milestone in this regard.”

In the paper

“GPT-3 samples still sometimes repeat themselves semantically at the document level”

In the paper

“We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA.”

In the paper

“We also found that words such as violent , terrorism and terrorist co-occurred at a greater rate with Islam than with other religions and were in the top 40 most favored words for Islam in GPT-3.”

In the paper

“it retains the biases of the data it has been trained on”

In the paper

“While we did find significant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances of experimentation and no successful deployments since then.”

In the paper

“In the final sentence the model generates a plausible conjugation for the word 'screeg' (namely 'screeghed'), although the use of the word is slightly awkward ('screeghed at each other') despite being plausible in the sense that it could describe a toy sword fight.”

In the paper

“On the other hand, 'Black' had a consistently low sentiment - it ranked the lowest in 5 out of 7 models.”

In the paper

“Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important.”

In the paper

“We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language modeling benchmark.”

In the paper

“training the GPT-3 175B consumed several thousand petaflop/s-days of compute during pre-training, compared to tens of petaflop/s-days for a 1.5B parameter GPT-2 model (Figure 2.2).”

In the paper

“The overlap analysis flagged 29% of examples as contaminated, and observed a 3 percentage point absolute decrease (4% relative decrease) in performance on the clean subset.”

In the paper

“Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8).”

In the paper

“contradict themselves”

In the paper

“its decisions are not easily interpretable”

In the paper

“Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting ( ∼ 33% ), whereas GPT-3 itself shows signs of life on Round 3.”

In the paper

“This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).”

In the paper

“The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k truncation) they are not able to perform consistently without human feedback.”

In the paper

“One notable exception was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very small, with the clean subset scoring within 0.5% of the full dataset.”

In the paper

“Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question.”

In the paper

“On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large.”

In the paper

“occasionally contain non-sequitur sentences or paragraphs”

In the paper

“even with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or only a few cents in energy costs.”

In the paper

“Another limitation broadly shared by language models is poor sample efficiency during pre-training.”

In the paper

“Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reflect stereotypes present in their training data.”

In the paper

“Zero-shot and one-shot performance is a few points behind, with the gains to few-shot being largest for bigger models.”

In the paper

“Related to this, the symbol insertion task shows high overlap but no effect on performance - this is because that task involves removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to many spurious matches.”

In the paper

“GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models.”

In the paper

“Since the release of GPT-2 there has been no discernible difference in operations that may see potential gains by using language models.”

In the paper

“We found 25% of the examples in the WMT16 German-English test set were marked as potentially contaminated, with an associated total effect size of 1-2 BLEU.”

In the paper

“Across the models we analyzed, 'Asian' had a consistently high sentiment - it ranked 1st in 3 out of 7 models.”

In the paper

“we have noticed informally that GPT-3 seems to have special difficulty with 'common sense physics', despite doing well on some datasets (such as PIQA [BZB + 19]) that test this domain.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites BZB + 19

In the paper

“First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks.”

In the paper

“All other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both.”

In the paper

“In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and sheriff.”

In the paper

“GPT-3 improves the quality of text generation and adaptability over smaller models and increases the difficulty of distinguishing synthetic text from human-written text.”

In the paper

“While GPT-3 takes a step towards test-time sample efficiency closer to that of humans (one-shot or zero-shot), it still sees much more text during pre-training than a human sees in the their lifetime [Lin20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Lin20

In the paper

“The overlap analysis flagged 45% of examples, and found a 2.6% decrease in performance on the clean subset.”

In the paper

“it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks”

In the paper

“Upon inspection, none of the flagged examples contain paired sentences resembling NMT training data and collisions were monolingual matches mostly of snippets of events discussed in the news.”

In the paper

“Mean participant age was ∼ 38 years old.”

In the paper

“We confirmed that every matching training document contained only the source passage, and none of the questions and answers in the dataset.”

In the paper

“Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but there is no apparent bias towards improved or degraded performance.”

In the paper

“Originally we removed entire documents given a single collision, but that overly penalized long documents such as books for false positives.”

In the paper

“This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the curve can perhaps be detected in Figure 3.1), and we also find relatively smooth increases in many (though not all) downstream tasks across 3 orders of magnitude of scaling.”

In the paper

“Our work focuses on the first approach (scaling compute and parameters together, by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ this strategy.”

In the paper

“Overall this decreased dataset size by an average of 10%.”

In the paper

“We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of state-of-the-art fine-tuned systems, as well as generating high-quality samples and strong qualitative performance at tasks defined on-the-fly.”

In the paper

“GPT-3 175B | 3.64E+03 | 3.14E+23 | 174,600 | 300 | 6 | 3 | 2 | 1.0 |”

In the paper

“The main disadvantage is that results from this method have so far been much worse than state-of-the-art fine-tuned models.”

In the paper

“This could also explain the comparatively low scores for RTE and CB, which also follow this format.”

In the paper

“there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a smooth trend of improvement with scale [KMH + 20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [KMH + 20]

In the paper

“As found in [KMH + 20, MKAT18], larger models can typically use a larger batch size, but require a smaller learning rate.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [KMH + 20, MKAT18]

In the paper

“for example [RWC + 19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind the state of the art.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [RWC + 19]

In the paper

“We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4).”

In the paper

“Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.”

In the paper

“Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions and/or the style of their answers are out-of-distribution for GPT-3.”

In the paper

“Previous work [KMH + 20] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [KMH + 20]

In the paper

“For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [SBBC19] is 94.0%.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites SBBC19

In the paper

“Similar to WebQS, the large gain from zero-shot to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to TriviaQA and WebQS.”

In the paper

“perhaps suggesting that larger models are more proficient meta-learners.”

In the paper

“The main advantages of few-shot are a major reduction in the need for task-specific data and reduced potential to learn an overly narrow distribution from a large but narrow fine-tuning dataset.”

In the paper

“Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk.”

In the paper

“In particular, the questions in NQs tend towards very fine-grained knowledge on Wikipedia specifically which could be testing the limits of GPT-3's capacity and broad pretraining distribution.”

In the paper

“Previous work [KMH + 20] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [KMH + 20]

In the paper

“Recent work [RWC + 19] attempts to do this via what we call 'in-context learning', using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [RWC + 19]

In the paper

“A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of models of this scale in their current form.”

In the paper

“Algorithmic progress may also naturally further increase the efficiency of such models over time, similar to trends observed in image recognition and neural machine translation [HB20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites HB20

In the paper

“We found a similar decrease in a 25x smaller model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias rather than memorization; examples which workers copied may simply be easier.”

In the paper

“Related work on language model detection by Ippolito et al. [IDCBE19] indicates that automatic discriminators like GROVER [ZHR + 19] and GLTR [GSR19] may have greater success at detecting model generated text than human evaluators.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [IDCBE19]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [ZHR + 19]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [GSR19]

In the paper

“Where GPT-3 is on this spectrum may also vary from task to task.”

In the paper

“Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question.”

In the paper

“GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above.”

In the paper

“Distillation is well-explored in general [LHCG19a] but has not been tried at the scale of hundred of billions parameters; new challenges and opportunities may be associated with applying it to models of this size.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LHCG19a

In the paper

“Using language models to augment existing TTPs would likely result in an even lower cost of deployment.”

In the paper

“If a social media disinformation bot produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the amount of human labor required in operating this bot. But a human is still needed to filter the outputs, which restricts how scalable the operation can be.”

In the paper

“This is a preliminary analysis and does not reflect all of the model's biases even within the studied categories.”

In the paper

“Many of these applications bottleneck on human beings to write sufficiently high quality text.”

In the paper

“Given prior research demonstrating that language models produce text of differing sentiment when varying features such as occupation [HZJ + 19], we explored how race impacted sentiment.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites HZJ + 19

In the paper

“A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks 'from scratch' at inference time, or if it simply recognizes and identifies tasks that it has learned during training.”

In the paper

“Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written.”

In the paper

“This is concerning, since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and producing demeaning portrayals amongst other potential harms [Cra17].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Cra17

In the paper

“Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling.”

In the paper

“On this task GPT-3 achieves 65.2% in the few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among college applicants was 57% [TL05] (random guessing yields 20%).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [TL05]

In the paper

“This may include fill-in-the-blank tasks, tasks that involve looking back and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then generating a very short answer.”

In the paper

“Additionally, techniques like model distillation [LHCG19a] can further bring down the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efficient versions of them for use in appropriate contexts.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LHCG19a

In the paper

“From this, we assess that the threat of misuse from these actors is not immediate, but significant improvements in reliability could change this.”

In the paper

“We conclude that either our conservative method substantially overestimated contamination or that contamination has little effect on performance.”

In the paper

“Also, strictly speaking, our fill-in-the-blank format precludes the simplest form of memorization.”

In the paper

“[RRS20] demonstrate benefits of customizing prediction to entities of interest.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites RRS20

In the paper

“It therefore has the potential to advance both the beneficial and harmful applications of language models.”

In the paper

“Finally, large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world [BHT + 20].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites BHT + 20

In the paper

“These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress.”

In the paper

“Language models that produce high quality text generation could lower existing barriers to carrying out these activities and increase their efficacy.”

In the paper

“For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a different approach is likely to be necessary.”

In the paper

“Synthetic tasks such as wordscrambling or defining nonsense words seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although possibly from data that is very different in organization and style than the test data.”

In the paper

“Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright remove problematic results, depending on the severity.”

In the paper

“As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising.”

In the paper

“Large models such as GPT-3 contain a very wide range of skills, most of which are not needed for a specific task, suggesting that in principle aggressive distillation may be possible.”

In the paper

“Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%).”

In the paper

“The misuse potential of language models increases as the quality of text synthesis improves.”

In the paper

“Any socially harmful activity that relies on generating text could be augmented by powerful language models.”

In the paper

“Ippolito et al. [IDCBE19] also note that human accuracy at detecting model generated text increases as humans observe more tokens.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [IDCBE19]

In the paper

“In order to ensure that the sample of participants for each experiment quiz was unique, participants were not allowed to take part in an experiment more than once.”

In the paper

“In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz B.”

In the paper

“Participants were excluded due to internet check fails.”

In the paper

“To compare means on the different runs, we performed a two-sample t-test for independent groups for each model against the control.”

In the paper

“Model generation selection methods, exclusion criteria, and statistical tests mirror those of the previous experiments.”

In the paper

“GPT-3 adapts well to many tasks other than the ones explored in the main body of the paper.”

In the paper

“The 95% confidence intervals were estimated from the t-distribution of the sample mean.”

In the paper

“All data comes from the ground truth datasets in this section, and no samples from GPT-3 are included here.”

In the paper

“When plotting a regression line in the graph of average participant accuracy vs model size, we fit a power law of the form ax -b .”

In the paper

“We recruited 718 unique participants to take part in 6 experiments.”

In the paper

“[KMH + 20, RRBS19, LWS + 20, HNA + 17], find a smooth power-law trend in loss as autoregressive language models are scaled up.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites KMH + 20
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites RRBS19
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites LWS + 20

In the paper

“Each quiz consisted of 25 articles: half (12-13) were human written and half (12-13) were model generated: the articles with human written completions in quiz A had model generated completions in quiz B and vice versa.”

In the paper

“The order of quiz question was shuffled for each participant.”

In the paper

“97 participants were excluded for failing an internet check question, leaving a total of 621 participants: 343 male, 271 female, and 7 other.”

In the paper

“We first experimented with a few prompts, then generated four samples with no additional editing or selection (sampling at temperature 1 using nucleus sampling [HBFC19] with P = 0 . 9 ).”

In the paper

“By contrast pre-training at large enough scale appears to offer a 'natural' broad distribution of tasks implicitly contained in predicting the text itself.”