CoChat
Sign Up
PreprintCited by 11,035Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Open access
Raffel, Colin, Shazeer, Noam, Roberts, Adam, Lee, Katherine, Narang, Sharan, Matena, Michael, Zhou, Yanqi, Li, Wei +1 more·2019·arXiv
DOI 10.48550/arxiv.1910.10683arXiv 1910.10683
Open this paper in CoChat — free accountOpen PDF

Abstract

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces T5, a unified text-to-text framework that casts every NLP task as text-in/text-out, and uses it to systematically compare transfer-learning choices. Combining insights from that study with scale and the new C4 corpus yields state-of-the-art results on many benchmarks, including GLUE, SuperGLUE, SQuAD, and CNN/Daily Mail.

Key findings

  • The text-to-text framework works across generative, classification, and regression tasks; T5-11B achieved a GLUE average of 90.3, SuperGLUE average of 88.9, SQuAD EM of 91.26, and CNN/Daily Mail ROUGE-2 of 21.55, reaching state of the art on 18 of 24 tasks considered.
  • Denoising objectives generally outperformed language modeling and deshuffling; an encoder-decoder architecture performed best, though sharing encoder/decoder parameters performed nearly as well, and span corruption with mean span length 3 slightly outperformed i.i.d. corruption on most non-translation benchmarks.
  • Scaling helped consistently: training longer, using larger models, and ensembling all improved performance; the 11B model performed best across tasks, and T5-Base substantially outperformed a baseline trained on the same 1T tokens, indicating non-scaling changes also mattered.
  • C4 filtering mattered: unfiltered C4 performed worst in every task, while in-domain pre-training data sometimes helped specific downstream tasks; repeating a small pre-training data set many times degraded performance.

Methods

  • The paper is primarily an empirical survey using a “coordinate ascent” approach: start from a baseline encoder-decoder Transformer and vary one factor at a time, including architecture, unsupervised objective, pre-training data, training strategy, and scaling.
  • The baseline model has about 220M parameters, is pre-trained for 2^19 steps on C4, fine-tuned for 2^18 steps, and evaluated on GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT English-to-German/French/Romanian.
  • C4 is built from April 2019 Common Crawl with heuristic cleaning, producing about 750 GB of English text; final T5 variants range from 60M to 11B parameters, are pre-trained on about 1T tokens with multi-task pre-training, and use beam search for translation and summarization.

Limitations

  • The paper states that its coordinate-ascent approach may miss second-order effects, that it cannot cover every transfer-learning technique, and that its baseline cannot be directly compared to BERT because it is an encoder-decoder model pre-trained for roughly one-quarter as many steps.
  • It reports high inter-run variance on low-resource tasks such as CoLA, CB, and COPA; notes it did not achieve state of the art on WMT translation, partly due to English-only unlabeled data and lack of backtranslation; and says SQuAD test-server resources were insufficient for its largest models.
  • The text also makes evident that the work depends on very large-scale TPU computation and that English-only pre-training limits multilingual applicability; the paper itself notes that ROUGE improvements may not correspond to more coherent summaries and that some reading-comprehension metrics may be biased toward machine predictions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.10683

Cites

162 of 162 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

37 of 57 cited sources checked · 20 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 2.4 Input and Output Format · p. 10

    Omitting results on the WNLI validation set is standard practice due to it being adversarial with respect to the training set.

    “Omitting results on the WNLI validation set is standard practice (Devlin et al., 2018) due to the fact that it is 'adversarial' with respect to the training set, i.e. validation examples are all slightly-perturbed versions of training examples with the opposite label.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Devlin et al., 2018 source

      Appendix B.1 (p. 15) says BERT excluded WNLI from its reported results, citing dataset-construction issues, poor performance by submitted systems, and fairness to OpenAI GPT. It does not establish that omission was standard practice or say that validation examples are perturbed training examples with opposite labels. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Findingcitation
    § 3.2.4 Results · p. 19

    Using a denoising objective always results in better downstream task performance compared to a language modeling objective.

    “Finally, we confirm the widely-held conception that using a denoising objective always results in better downstream task performance compared to a language modeling objective.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Devlin et al., 2018 source

      Section 5.1 (p. 8) reports that BERT’s masked-language-model variant outperformed its left-to-right language-model variant on every task tested. That supports the narrower comparison, but not the claim that a denoising objective always improves downstream performance: the paper evaluates particular models and tasks, and Section 3.1 (p. 4) calls its objective a masked language model rather than a denoising auto-encoder. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Voita et al., 2019 source

      The Introduction (p. 2) proposes a hypothesis for why masked language modeling (MLM) may be preferable to language modeling (LM) for pretraining, and the Conclusions (p. 9) call it a possible explanation for MLM’s superior performance. The paper does not establish that a denoising objective always produces better downstream-task performance; the claim turns a qualified explanation into a universal result. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Lample and Conneau, 2019 source

      Section 5.3 (pp. 6–7) reports that masked language modeling (MLM) pretraining outperformed causal language modeling (CLM) pretraining in the machine-translation comparisons studied. It does not establish that a denoising objective always yields better performance across downstream tasks; the paper also distinguishes MLM pretraining from the denoising auto-encoding loss used during translation training. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data
    § 3.7 Putting It All Together · p. 38

    T5 outperformed the previous state-of-the-art (ALBERT) by over one point on the SQuAD Exact Match score.

    “For SQuAD, we outperformed the previous state-of-the-art (ALBERT (Lan et al., 2019)) by over one point on the Exact Match score.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lan et al., 2019 source
  • Findingcitation
    § 3.7 Putting It All Together · p. 36

    The baseline model uses a relatively small amount of pre-training compared to BERT, XLNet, and RoBERTa.

    “Our baseline model uses a relatively small amount of pre-training ( 1 / 4 as much as BERT (Devlin et al., 2018), 1 / 16 as much as XLNet (Yang et al., 2019), 1 / 64 as much as RoBERTa (Liu et al., 2019c), etc.).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Devlin et al., 2018 source

      Appendix A.2 (p. 13) reports BERT’s pre-training amount: 1,000,000 steps at 128,000 tokens per batch. The paper does not describe the citing authors’ baseline, so it cannot establish that the baseline used one-quarter as much pre-training as BERT. The XLNet and RoBERTa comparisons are attributed to other citations. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Yang et al., 2019 source

      Section 3.1 (p. 6) reports that XLNet-Large was pretrained for 500,000 steps with a batch size of 8,192, supporting the cited figure as an XLNet pretraining benchmark. The paper does not describe the citing authors’ baseline, so it cannot establish that the baseline used 1/16 as much pretraining as XLNet. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Liu et al., 2019c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § Abstract · p. 1

    The paper achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

    “By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.”
    Quote in the paperSubstantial claim
  • May combine two claims. 48
  • Classified as probably not a substantial claim. 54
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation
    § 2.2 The Colossal Clean Crawled Corpus · p. 6

    Prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope.

    “However, we opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data (Zellers et al., 2019; Liu et al., 2019c), comprise only Creative Commons content (Habernal et al., 2016), or are focused on parallel training data for machine translation (Smith et al., 2013)).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Zellers et al., 2019 source
    • cites Liu et al., 2019c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Habernal et al., 2016 source

  • May combine two claims.

    Findingown data
    § 3.1.2 Training · p. 12

    A triangular learning rate produced slightly better results than the inverse square root schedule but requires knowing the total number of training steps ahead of time.

    “We also experimented with using a triangular learning rate (Howard and Ruder, 2018), which produced slightly better results but requires knowing the total number of training steps ahead of time.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Howard and Ruder, 2018 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.1 Model · p. 4

    The Transformer was initially shown to be effective for machine translation, but has subsequently been used in a wide variety of NLP settings.

    “The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings (Radford et al., 2018; Devlin et al., 2018; McCann et al., 2018; Yu et al., 2018).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Radford et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Devlin et al., 2018 source
    • cites McCann et al., 2018 source

      Further Related Work (p. 17) says attention-only models were effective for machine translation. The MQAN section (pp. 5–6) describes using Transformer-derived self-attention layers across decaNLP tasks, but the model also uses recurrent and coattentive layers; it does not establish that the Transformer architecture itself was used across those settings. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3.4 Corrupting Spans · p. 23

    Corrupting spans was previously considered as a pre-training objective for BERT, where it was found to improve performance.

    “Corrupting spans was also previously considered as a pre-training objective for BERT, where it was found to improve performance (Joshi et al., 2019).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Joshi et al., 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.4 Unsupervised Objective · p. 12

    Preliminary work that applied transfer learning to NLP used a causal language modeling objective for pre-training.

    “Preliminary work that applied the transfer learning paradigm of pre-training and fine-tuning all of the model's parameters to NLP problems used a causal language modeling objective for pre-training (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Dai and Le, 2015 source
    • cites Peters et al., 2018 source

      Section 3.1 describes pre-training with forward and backward language models that predict tokens from preceding or following context, respectively. But Section 3.3 says the pre-trained biLM weights are frozen when ELMo is added to supervised task models. Peters et al. therefore support the language-modeling objective, not the attributed fine-tuning of all model parameters. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
    • cites Radford et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1.2 Training · p. 11

    BERT used roughly 137B tokens and RoBERTa used roughly 2.2T tokens for pre-training.

    “This is considerably less than BERT (Devlin et al., 2018), which used roughly 137B tokens, or RoBERTa (Liu et al., 2019c), which used roughly 2 . 2T tokens.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Devlin et al., 2018 source
    • cites Liu et al., 2019c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § 3.1.4 Unsupervised Objective · p. 12

    Denoising objectives produce better performance than causal language modeling and have become standard.

    “However, it has recently been shown that 'denoising' objectives (Devlin et al., 2018; Taylor, 1953) (also called 'masked language modeling') produce better performance and as a result they have quickly become standard.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Devlin et al., 2018 source

      Section 4.1 (p. 6) reports that BERT outperformed the left-to-right OpenAI GPT model on the GLUE tasks, supporting the performance comparison. Section 3.1 (p. 4) identifies BERT’s masked-language-model objective, but the paper does not establish that such objectives had become standard; it also notes that BERT’s training differs from GPT’s in other ways (Appendix A.4, p. 14). (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Taylor, 1953 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction · p. 2

    Unsupervised pre-training for NLP has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks.

    “This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al., 2018; Yang et al., 2019; Dong et al., 2019; Liu et al., 2019c; Lan et al., 2019).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Devlin et al., 2018 source
    • cites Yang et al., 2019 source
    • cites Dong et al., 2019 source
    • Rests on an assumption: Unlabeled text data is available en masse thanks to the Internet.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4.1 Unlabeled Data Sets · p. 26

    Prior work has made similar observations; Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.

    “Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Beltagy et al. (2019) source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.6 Scaling · p. 34

    General methods that can leverage additional computation ultimately win out against methods that rely on human expertise.

    “The 'bitter lesson' of machine learning research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise (Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b; Keskar et al., 2019a).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Sutton, 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hestness et al., 2017 source

      The Introduction (p. 1) describes scaling data and computation as a lower-risk complement to creative model-architecture search, and the Conclusion (p. 13) says the resulting accuracy improvements are empirically predictable. The paper does not claim that computation-leveraging methods ultimately win over methods relying on human expertise. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Shazeer et al., 2017 source

      Section 1.1 (p. 1) says that increasing neural-network capacity can improve prediction accuracy, and the Abstract (p. 1) reports improved benchmark results from a much larger model. The paper does not compare general methods with methods relying on human expertise or claim that the former ultimately win out. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § 3.5.2 Multi-task Learning · p. 31

    An extremely important factor in multi-task learning is how much data from each task the model should be trained on.

    “As pointed out by Arivazhagan et al. (2019), an extremely important factor in multi-task learning is how much data from each task the model should be trained on.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Arivazhagan et al. (2019) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.2 Multi-task Learning · p. 32

    The multi-task setup can confer benefits across very similar tasks.

    “though it has been shown that the multi-task setup can confer benefits across very similar tasks Liu et al. (2019b); Ratner et al. (2018).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Liu et al. (2019b)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ratner et al. (2018) source

  • May combine two claims.

    Findingcitation
    § 3.5.1 Fine-tuning Methods · p. 30

    Lower-resource tasks like SQuAD work well with a small value of d, whereas higher resource tasks require a large dimensionality to achieve reasonable performance.

    “Pursuant with past results (Houlsby et al., 2019; Bapna et al., 2019) we find that lower-resource tasks like SQuAD work well with a small value of d whereas higher resource tasks require a large dimensionality to achieve reasonable performance.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Houlsby et al., 2019 source

      Section 3.5 (p. 6) reports that SQuAD performs well with adapters as small as size 2. But the paper does not identify SQuAD as a lower-resource task or establish that higher-resource tasks require a larger dimension. Section 3.6 (p. 7) instead reports stable performance across adapter sizes 8, 64, and 256 on its classification tasks. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Bapna et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4.2 Pre-training Data set Size · p. 28

    Baevski et al. (2019) similarly observed that truncating the pre-training data set size can degrade downstream task performance.

    “Baevski et al. (2019) similarly observed that truncating the pre-training data set size can degrade downstream task performance.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Baevski et al. (2019) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.3 Combining Multi-Task Learning with Fine-Tuning · p. 32

    MT-DNN achieved state-of-the-art performance on GLUE and other benchmarks when it was introduced.

    “This is the method used by the 'MT-DNN' (Liu et al., 2015, 2019b), which achieved state-of-the-art performance on GLUE and other benchmarks when it was introduced.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Liu et al., 2015 source

      The available passages are bibliographic metadata for the 2015 paper (pp. 912–921), not its results. Searches for GLUE, MT-DNN, and state-of-the-art benchmark performance returned no passage addressing the claim, so our copy cannot verify it. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites Liu et al., 2019b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.6 Scaling · p. 34

    Scaling up produces improved performance compared to more carefully-engineered methods in transfer learning for NLP.

    “Recent results suggest that this may hold true for transfer learning in NLP (Liu et al., 2019c; Radford et al., 2019; Yang et al., 2019; Lan et al., 2019), i.e. it has repeatedly been shown that scaling up produces improved performance compared to more carefully-engineered methods.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Liu et al., 2019c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Radford et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yang et al., 2019 source

      Section 3.1 (p. 6) says XLNet scaled up training, and Section 3.3 (p. 7) reports that it generally outperformed BERT and RoBERTa. But the paper also attributes gains to XLNet’s novel pretraining objective and architectural changes (Introduction, p. 2); it does not show that scaling up, rather than those engineered changes, produced better performance than more carefully engineered methods. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.2 Multi-task Learning · p. 32

    The finding that a multi-task model is outperformed by separate models trained on each individual task has previously been observed by Arivazhagan et al. (2019) and McCann et al. (2018).

    “The finding that a multi-task model is outperformed by separate models trained on each individual task has previously been observed e.g. by Arivazhagan et al. (2019) and McCann et al. (2018)”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Arivazhagan et al. (2019) source

      Section 4.1 (p. 6) reports that multilingual models perform worse than separate bilingual baselines on high-resource language pairs. But it also reports that joint training beats those baselines on low-resource pairs, so the paper does not support an unqualified claim that separate models outperform the multi-task model on each task. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites McCann et al. (2018) source

      Appendix D (pp. 22–23) reports that fully joint training struggles to reach single-task performance on several tasks and that a gap remains. But the Introduction (p. 2) says the multitask MQAN, with the right training strategy, achieves performance comparable to ten separately trained MQANs; Related Work (p. 9) says it performs nearly as well or better on each task. The claim overstates the paper’s finding as a general result that separate models outperform the multitask model. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S1 (asserted).

    Premisecitation
    § 3.5.1 Fine-tuning Methods · p. 29

    It has been argued that fine-tuning all of the model's parameters can lead to suboptimal results, particularly on low-resource tasks.

    “It has been argued that fine-tuning all of the model's parameters can lead to suboptimal results, particularly on low-resource tasks (Peters et al., 2019).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Peters et al., 2019 source

      The Conclusion (page unavailable) says that fine-tuning can perform differently from feature extraction depending on how similar the pretraining and target tasks are; the Impact of additional parameters section also describes fine-tuning underperforming in dissimilar settings. The cited paper does not identify low-resource tasks as a particular reason that tuning all parameters yields suboptimal results. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.6 Scaling · p. 34

    Training with a 4× larger batch size can yield a different outcome than training for 4× as many steps.

    “However, training with a 4 × larger batch size can yield a different outcome than training for 4 × as many steps (Shallue et al., 2018).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Shallue et al., 2018 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.1 Fine-tuning Methods · p. 29

    Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier fed sentence embeddings from a fixed pre-trained model.

    “Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model (Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Subramanian et al., 2018 source
    • cites Kiros et al., 2015 source
    • cites Logeswaran and Lee, 2018 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4 Pre-training Data set · p. 24

    Some recent notable exceptions have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).

    “Some recent notable exceptions (Baevski et al., 2019; Liu et al., 2019c; Yang et al., 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Baevski et al., 2019 source
    • cites Liu et al., 2019c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yang et al., 2019 source

  • May combine two claims.

    Findingnone given
    § 2.2 The Colossal Clean Crawled Corpus · p. 6

    The majority of Common Crawl's web extracted text is not natural language; it largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text.

    “Unfortunately, the majority of the resulting text is not natural language. Instead, it largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.4 Input and Output Format · p. 8

    Changing the exact wording of the task prefix had limited impact.

    “Note that the choice of text prefix used for a given task is essentially a hyperparameter; we found that changing the exact wording of the prefix had limited impact and so did not perform extensive experiments into different prefix choices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2 The Colossal Clean Crawled Corpus · p. 7

    C4 is about 750 GB.

    “This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2 The Colossal Clean Crawled Corpus · p. 7

    C4 is orders of magnitude larger than most data sets used for pre-training.

    “This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2.2 The Colossal Clean Crawled Corpus · p. 7

    C4 comprises reasonably clean and natural English text.

    “This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.4 Input and Output Format · p. 8

    The model never output text on a text classification task that did not correspond to any possible label in any of the trained models.

    “In this case, we always count the model's output as wrong, though we never observed this behavior in any of our trained models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 3.1.5 Baseline Performance · p. 14

    The only exception to pre-training gains is WMT English to French, which is a large enough data set that gains from pre-training tend to be marginal.

    “The only exception is WMT English to French, which is a large enough data set that gains from pre-training tend to be marginal.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3.1 Disparate High-Level Approaches · p. 22

    The 'drop corrupted tokens' objective achieved scores of 84.44 GLUE, 19.31 CNNDM, 80.52 SQuAD, 68.67 SGLUE, 27.07 EnDe, 39.76 EnFr, and 27.82 EnRo.

    “Drop corrupted tokens | 84 . 44 | 19 . 31 | 80 . 52 | 68 . 67 | 27 . 07 | 39 . 76 | 27 . 82”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.2.2 Comparing Different Model Structures · p. 18

    In practice, nearly identical step times were observed for L-layer language models versus L+L-layer encoder-decoder models, suggesting a roughly equivalent computational cost.

    “In practice, however, we observed nearly identical step times for L -layer language models versus L + L -layer encoder-decoder models, suggesting a roughly equivalent computational cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.1.5 Baseline Performance · p. 14

    Training without pre-training for the same number of fine-tuning steps achieved scores of 66.22 on GLUE, 17.60 on CNNDM, 50.31 on SQuAD, 53.04 on SGLUE, 25.86 on EnDe, 39.77 on EnFr, and 24.04 on EnRo.

    “No pre-training | 66 . 22 | 17 . 60 | 50 . 31 | 53 . 04 | 25 . 86 | 39 . 77 | 24 . 04”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 3.3.1 Disparate High-Level Approaches · p. 22

    The 'drop corrupted tokens' objective achieved a GLUE score of 84.44, higher than 'replace corrupted spans' (83.28) and BERT-style (82.96).

    “Drop corrupted tokens | 84 . 44 | 19 . 31 | 80 . 52 | 68 . 67 | 27 . 07 | 39 . 76 | 27 . 82”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3.4 Corrupting Spans · p. 24

    There was a limited difference between the span-corruption objective and the i.i.d-corruption objective, though average span length 10 slightly underperforms other values in some cases.

    “Again, we find a limited difference between these objectives, though the version with an average span length of 10 slightly underperforms the other values in some cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.3.1 Disparate High-Level Approaches · p. 22

    The 'replace corrupted spans' objective achieved scores of 83.28 GLUE, 19.24 CNNDM, 80.88 SQuAD, 71.36 SGLUE, 26.98 EnDe, 39.82 EnFr, and 27.65 EnRo.

    “⋆ Replace corrupted spans | 83 . 28 | 19 . 24 | 80 . 88 | 71 . 36 | 26 . 98 | 39 . 82 | 27 . 65”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 3.1.5 Baseline Performance · p. 14

    BERT BASE achieved an exact match score of 80.8 on SQuAD and an accuracy of 84.4 on MNLI-matched, whereas the baseline achieved 80.88 and 84.24, respectively.

    “BERT BASE achieved an exact match score of 80 . 8 on SQuAD and an accuracy of 84 . 4 on MNLI-matched, whereas we achieve 80 . 88 and 84 . 24, respectively”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.1.5 Baseline Performance · p. 14

    The baseline model achieved average validation scores of 83.28 on GLUE, 19.24 on CNNDM, 80.88 on SQuAD, 71.36 on SGLUE, 26.98 on EnDe, 39.82 on EnFr, and 27.65 on EnRo.

    “⋆ Baseline average | 83 . 28 | 19 . 24 | 80 . 88 | 71 . 36 | 26 . 98 | 39 . 82 | 27 . 65”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 3.1.2 Training · p. 11

    2^35 tokens covers only a fraction of the entire C4 data set, so no data was repeated during pre-training.

    “Note that 2 35 tokens only covers a fraction of the entire C4 data set, so we never repeat any data during pre-training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 3.1.5 Baseline Performance · p. 14

    The baseline model's standard deviations across runs were 0.235 on GLUE, 0.065 on CNNDM, 0.343 on SQuAD, 0.416 on SGLUE, 0.112 on EnDe, 0.090 on EnFr, and 0.108 on EnRo.

    “Baseline standard deviation | 0 . 235 | 0 . 065 | 0 . 343 | 0 . 416 | 0 . 112 | 0 . 090 | 0 . 108”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.2.4 Results · p. 19

    The shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial.

    “We also note that the shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.

  • May combine two claims.

    Findingnone given
    § 3.2.4 Results · p. 19

    The encoder-decoder denoising variant has the highest parameter count (2P) but the same computational cost as P-parameter decoder-only models.

    “This variant has the highest parameter count (2 P ) but the same computational cost as the P -parameter decoder-only models.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.

  • May combine two claims.

    Findingown data
    § 3.3.1 Disparate High-Level Approaches · p. 21

    The BERT-style objective performs best among the three disparate pre-training objectives, though the prefix language modeling objective attains similar performance on the translation tasks.

    “Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.4.1 Unlabeled Data Sets · p. 26

    Using the Wikipedia + TBC corpus produced a SuperGLUE score of 73.24, beating the baseline C4 score of 71.36.

    “For example, using the Wikipedia + TBC corpus produced a SuperGLUE score of 73 . 24, beating our baseline's score (using C4) of 71 . 36.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.5.1 Fine-tuning Methods · p. 30

    Gradual unfreezing caused a minor degradation in performance across all tasks, though it provided some speedup during fine-tuning.

    “We found that gradual unfreezing caused a minor degradation in performance across all tasks, though it did provide some speedup during fine-tuning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 3.5.2 Multi-task Learning · p. 32

    Temperature-scaled mixing provides a means of obtaining reasonable performance from most tasks, with T = 2 performing the best in most cases.

    “Finally, we note that temperature-scaled mixing also provides a means of obtaining reasonable performance from most tasks, with T = 2 performing the best in most cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.4.1 Unlabeled Data Sets · p. 26

    Removing the heuristic filtering from C4 uniformly degrades performance and makes the unfiltered variant perform the worst in every task.

    “A first obvious takeaway is that removing the heuristic filtering from C4 uniformly degrades performance and makes the unfiltered variant perform the worst in every task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 42

    The authors trained substantially larger models (up to 11 billion parameters) and achieved state-of-the-art results across many of the benchmarks they considered.

    “We combined our above insights and trained substantially larger models (up to 11 billion parameters) to achieve state-of-the-art results across many of the benchmarks we considered.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 42

    Each of training on more data, training a larger model, and using an ensemble of models conferred a significant boost in performance.

    “We found each approach conferred a significant boost in performance, though training a smaller model on more data was often outperformed by training a larger model for fewer steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 41

    The text-to-text framework obtained comparable performance to task-specific architectures and ultimately produced state-of-the-art results when combined with scale.

    “In spite of its simplicity, we found the text-totext framework obtained comparable performance to task-specific architectures and ultimately produced state-of-the-art results when combined with scale.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.6 Scaling · p. 35

    There was no clear winner between training for 4× as many steps or using a 4× larger batch size, though both were beneficial.

    “There was no clear winner between training for 4 × as many steps or using a 4 × larger batch size, though both were beneficial.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 41

    Sharing the parameters in the encoder and decoder did not result in a substantial performance drop while halving the total parameter count.

    “We also showed that sharing the parameters in the encoder and decoder did not result in a substantial performance drop while halving the total parameter count.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2 Outlook · p. 43

    Larger models tend to perform better.

    “An unsurprising but important result from our study is that larger models tend to perform better.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 42

    In the authors' experiments, updating all of a pre-trained model's parameters during fine-tuning outperformed methods designed to update fewer parameters, although updating all parameters is most expensive.

    “We found that the basic approach of updating all of a pre-trained model's parameters during fine-tuning outperformed methods that are designed to update fewer parameters, although updating all parameters is most expensive.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.6 Scaling · p. 35

    Increasing the training time and/or model size consistently improves the baseline.

    “Unsurprisingly, increasing the training time and/or model size consistently improves the baseline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 3.7 Putting It All Together · p. 38

    In terms of parameter count, the T5-11B model variant is the largest model that has been submitted to the GLUE benchmark.

    “In terms of parameter count, our 11B model variant is the largest model that has been submitted to the GLUE benchmark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Takeaways · p. 41

    The text-to-text framework can be successfully applied to generative tasks, classification tasks, and regression tasks.

    “We showed how this approach can be successfully applied to generative tasks like abstractive summarization, classification tasks like natural language inference, and even regression tasks like STS-B.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.7 Putting It All Together · p. 40

    T5 models generate coherent and largely correct summaries.

    “Despite these potential issues, we find that our models do generate coherent and largely correct summaries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Appendix C. Example Predictions on CNN/Daily Mail · p. 45

    The T5-11B model generates fluent summaries on CNN/Daily Mail validation examples.

    “To show that our model is generating fluent summaries, we include a few example decodes from our best model (T5-11B) on the validation set along with the ground-truth summaries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.7 Putting It All Together · p. 40

    On CNN/Daily Mail, T5 attains state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score.

    “Finally, on CNN/Daily Mail we attain state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 2.4 Input and Output Format · p. 10

    The paper never trains on WNLI and never reports results on the WNLI validation set to avoid leaking validation examples into training data.

    “To avoid leaking validation examples into our training data (a particular issue in the multi-task experiments of Section 3.5.2), we therefore never train on WNLI and never report results on the WNLI validation set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 2.4 Input and Output Format · p. 10

    The paper does not include WNLI in the average GLUE score whenever it reports on the validation set.

    “As such, we do not include WNLI in the average GLUE score whenever we report on the validation set (all sections except Section 3.7 where results are presented on the test sets).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.1.2 Training · p. 12

    For all experiments except those in Section 3.7, results are reported on the validation set to avoid performing model selection on the test set.

    “For all of the experiments except those in Section 3.7, we report results in the validation set to avoid performing model selection on the test set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.3.3 Varying the Corruption Rate · p. 23

    Using a larger corruption rate results in longer targets, which can potentially slow down training.

    “Using a larger corruption rate also results in longer targets, which can potentially slow down training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 3.2.2 Comparing Different Model Structures · p. 18

    An encoder-decoder model with L layers in the encoder and L layers in the decoder has approximately the same number of parameters as a language model with 2L layers, but approximately the same computational cost as a language model with only L layers.

    “To see why, first note an encoder-decoder model with L layers in the encoder and L layers in the decoder has approximately the same number of parameters as a language model with 2 L layers. However, the same L + L encoder-decoder model will have approximately the same computational cost as a language model with only L layers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.1.5 Baseline Performance · p. 13

    The baseline model was trained 10 times from scratch with different random initializations and data set shuffling.

    “As a cheaper alternative, we train our baseline model 10 times from scratch (i.e. with different random initializations and data set shuffling)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 2.4 Input and Output Format · p. 9

    The text-to-text framework allows for generative tasks like machine translation and abstractive summarization where it is not possible to enumerate all possible output choices.

    “In contrast, our framework also allows for generative tasks like machine translation and abstractive summarization where it is not possible to enumerate all possible output choices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.3.2 Simplifying the BERT Objective · p. 23

    The variants that do not require predicting the full original sequence make target sequences shorter and consequently make training faster.

    “The two variants that do not require predicting the full original sequence ('replace corrupted spans' and 'drop corrupted spans') are both potentially attractive since they make the target sequences shorter and consequently make training faster.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.1.5 Baseline Performance · p. 14

    The baseline cannot be directly compared to BERT BASE because the baseline is an encoder-decoder model and was pre-trained for roughly 1/4 as many steps.

    “Note that we cannot directly compare our baseline to BERT BASE because ours is an encoder-decoder model and was pre-trained for roughly 1 / 4 as many steps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3.1.2 Training · p. 12

    A checkpoint was saved every 5,000 steps and results were reported on the model checkpoint corresponding to the highest validation performance.

    “We save a checkpoint every 5 , 000 steps and report results on the model checkpoint corresponding to the highest validation performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.1.2 Training · p. 12

    The fine-tuning step count of 2^18 was chosen as a trade-off between high-resource tasks, which benefit from additional fine-tuning, and low-resource tasks, which overfit quickly.

    “This value was chosen as a trade-off between the high-resource tasks (i.e. those with large data sets), which benefit from additional fine-tuning, and low-resource tasks (smaller data sets), which overfit quickly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 3.1.2 Training · p. 11

    Using only 2^35 tokens results in a reasonable computational budget while still providing sufficient pre-training for acceptable performance.

    “Using only 2 35 tokens results in a reasonable computational budget while still providing a sufficient amount of pre-training for acceptable performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.1.2 Training · p. 12

    For models fine-tuned on multiple tasks, the best checkpoint was chosen for each task independently.

    “For models fine-tuned on multiple tasks, we choose the best checkpoint for each task independently.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 3.1.4 Unsupervised Objective · p. 12

    The choices to mask consecutive spans of tokens and only predict dropped-out tokens were made to reduce the computational cost of pre-training.

    “Our choices to mask consecutive spans of tokens and only predict dropped-out tokens were made to reduce the computational cost of pre-training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductreasoning
    § 3. Experiments · p. 10

    The coordinate ascent approach might miss second-order effects.

    “This 'coordinate ascent' approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction · p. 2

    Modern techniques for transfer learning in NLP often pre-train using unsupervised learning on unlabeled data.

    “In contrast, modern techniques for transfer learning in NLP often pre-train using unsupervised learning on unlabeled data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Unlabeled text data is available en masse thanks to the Internet.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.2 Multi-task Learning · p. 30

    Most applications of multi-task learning to NLP add task-specific classification networks or use different loss functions for each task.

    “In contrast, most applications of multi-task learning to NLP add task-specific classification networks or use different loss functions for each task (Liu et al., 2019b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liu et al., 2019b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § 3.5.3 Combining Multi-Task Learning with Fine-Tuning · p. 33

    Translation tasks benefit less from English pre-training, whereas unsupervised pre-training is an important factor in the other tasks.

    “This could suggest that the translation tasks benefit less from (English) pre-training, whereas unsupervised pre-training is an important factor in the other tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.3.4 Corrupting Spans · p. 24

    The span-corruption objective provides some speedup during training compared to the i.i.d. noise approach because span corruption produces shorter sequences on average.

    “Fortunately, the span-corruption objective also provides some speedup during training compared to the i.i.d. noise approach because span corruption produces shorter sequences on average.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 3.4.2 Pre-training Data set Size · p. 28

    This effect may be more pronounced for larger model sizes; a bigger model may be more prone to overfitting to a smaller pre-training data set.

    “We also note that this effect may be more pronounced for larger model sizes, i.e. a bigger model may be more prone to overfitting to a smaller pre-training data set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5.2 Multi-task Learning · p. 31

    A similar exploration of mixing proportions was performed by Wang et al. (2019a).

    “A similar exploration was performed by Wang et al. (2019a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al. (2019a)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 3.3.5 Discussion · p. 24

    Different objectives or parameterizations can lead to different sequence lengths and thus different training speeds.

    “However, different objectives (or parameterizations of objectives) can lead to different sequence lengths and thus different training speeds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4.1 Unlabeled Data Sets · p. 26

    Liu et al. (2019c) observed that pre-training on a more diverse data set yielded improvements on downstream tasks.

    “Liu et al. (2019c) also observed that pre-training on a more diverse data set yielded improvements on downstream tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liu et al. (2019c)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 4.1 Takeaways · p. 42

    The use of a large and diverse data set like C4 is motivated for generic language understanding tasks.

    “This motivates the use of a large and diverse data set like C4 for generic language understanding tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 4.1 Takeaways · p. 41

    An encoder-decoder model uses twice as many parameters as encoder-only or decoder-only architectures but has a similar computational cost.

    “Though an encoder-decoder model uses twice as many parameters as 'encoder-only' (e.g. BERT) or 'decoder-only' (language model) architectures, it has a similar computational cost.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Appendix B. Converting WNLI to Our Text-to-Text Format · p. 45

    The authors do not train on any of the data from WNLI.

    “Note that as discussed in Section 2.4, we do not train on any of the data from WNLI.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 3.7 Putting It All Together · p. 38

    The model with the highest performance on the SQuAD test set also reported results on the validation set, so comparison to the state-of-the-art is still possible.

    “Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set, so we can still compare to what is ostensibly the state-of-the-art.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The model with the highest performance on the SQuAD test set also reported results on the validation set.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.7 Putting It All Together · p. 38

    SQuAD results are reported on the validation set because the computational resources on the test server are insufficient for obtaining predictions from the largest models.

    “Unfortunately, the computational resources on this server are insufficient for obtaining predictions from our largest models. As a result, we instead continue to report performance on the SQuAD validation set.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The computational resources on the SQuAD test server are insufficient for obtaining predictions from the largest models.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data
    § 4.2 Outlook · p. 44

    The authors' results stem from the combination of a straightforward and unified text-to-text framework, their new C4 data set, and insights from their systematic study.

    “Our results stem from the combination of a straightforward and unified text-to-text framework, our new C4 data set, and insights from our systematic study.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given
    § Appendix C. Example Predictions on CNN/Daily Mail · p. 45

    The examples in Appendix C were selected at random and were not cherry-picked.

    “These examples selected at random and were not cherry-picked.”
    Quote in the paperNot a substantial claim
88 of 88

  • The paper relies on this without support.

    Common Crawl produces about 20TB of text data extracted from web pages each month.

    “the Common Crawl project 2 produces about 20TB of text data extracted from web pages each month.”

  • The paper relies on this without support.

    The architectural changes (removing Layer Norm bias, placing layer normalization outside the residual path, and using a different position embedding scheme) are orthogonal to the experimental factors considered in the empirical survey of transfer learning.

    “Since these architectural changes are orthogonal to the experimental factors we consider in our empirical survey of transfer learning, we leave the ablation of their impact for future work.”

  • The paper relies on this without support.

    Unlabeled text data is available en masse thanks to the Internet.

    “Beyond its empirical strength, unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet”

  • The authors state this as an assumption.

    The variance over the 10 baseline runs also applies to each experimental variant.

    “we train our baseline model 10 times from scratch (i.e. with different random initializations and data set shuffling) and assume that the variance over these runs of the base model also applies to each experimental variant.”

  • The paper relies on this without support.

    Most changes made in the experiments do not have a dramatic effect on the inter-run variance.

    “We don't expect most of the changes we make to have a dramatic effect on the inter-run variance, so this should provide a reasonable indication of the significance of different changes.”

  • The authors state this as an assumption.

    An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.

    “we make the simplifying assumption that an L + L -layer encoder-decoder model has the same number of parameters as an 2 L -layer language model.”

  • The paper relies on this without support.

    New pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code.

    “However, unlike objectives and benchmarks, new pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code.”

  • The paper relies on this without support.

    There has been relatively little comparison of different pre-training data sets and a lack of a standard data set used for pre-training.

    “As a result, there has been relatively little comparison of different pre-training data sets as well as a lack of a 'standard' data set used for pre-training.”

  • The paper relies on this without support.

    MultiRC's largest source of data comes from fiction books, which is exactly the domain covered by TBC.

    “MultiRC is a reading comprehension data set whose largest source of data comes from fiction books, which is exactly the domain covered by TBC.”

  • The paper relies on this without support.

    ReCoRD is a data set that measures reading comprehension on news articles.

    “ReCoRD, a data set that measures reading comprehension on news articles.”

  • The paper relies on this without support.

    SQuAD is a question-answering data set with passages sourced from Wikipedia.

    “SQuAD, which is a question-answering data set with passages sourced from Wikipedia.”

  • The paper relies on this without support.

    Obtaining additional unlabeled data is cheap and easy.

    “obtaining additional unlabeled data is cheap and easy”

  • The paper relies on this without support.

    There is potential for task interference or negative transfer, where achieving good performance on one task can hinder performance on another.

    “An additional issue is the potential for 'task interference' or 'negative transfer', where achieving good performance on one task can hinder performance on another.”

  • The paper relies on this without support.

    The pre-training objective is stochastic and can help prevent the model from seeing the same exact data multiple times.

    “because our pre-training objective is itself stochastic and can help prevent the model from seeing the same exact data multiple times.”

  • The paper relies on this without support.

    Leave-one-out multi-task training simulates the real-world setting where a pre-trained model is fine-tuned on a task it had not seen during pre-training.

    “This simulates the real-world setting where a pre-trained model is fine-tuned on a task it had not seen during pre-training.”

  • The paper relies on this without support.

    Multi-task pre-training provides a diverse mixture of supervised tasks.

    “Note that multi-task pre-training provides a diverse mixture of supervised tasks.”

  • The authors state this as an assumption.

    Larger models trained for longer might benefit from a larger proportion of unlabeled data because they are more likely to overfit to smaller training data sets.

    “We hypothesize that larger models trained for longer might benefit from a larger proportion of unlabeled data because they are more likely to overfit to smaller training data sets.”

  • The paper relies on this without support.

    The computational resources on the SQuAD test server are insufficient for obtaining predictions from the largest models.

    “Unfortunately, the computational resources on this server are insufficient for obtaining predictions from our largest models.”

  • The paper relies on this without support.

    The model with the highest performance on the SQuAD test set also reported results on the validation set.

    “Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set”

  • The paper relies on this without support.

    Humans achieve 100% accuracy on both COPA and WSC.

    “humans achieve 100% accuracy on both COPA and WSC”

  • The paper relies on this without support.

    C4 is so large that the authors do not complete one pass over the data even when training for 2^23 steps.

    “C4 is so large that we do not complete one pass over the data even when training for 2 23 steps.”

  • The paper relies on this without support.

    The primary concern in multi-task learning is setting the proportion of each task to train on.

    “The primary concern in multi-task learning is setting the proportion of each task to train on.”
22 of 22

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models are recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance.

    “LLMs refer to the recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance [9,16].”

    From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [16]

    Section 3.7 (p. 37) describes T5 variants with 2.8 billion and 11 billion parameters, and Section 4.1 (p. 42) reports state-of-the-art results across many benchmarks. Those results follow pre-training and fine-tuning; the cited passages do not establish strong few-shot or zero-shot performance or define LLMs as models exceeding a billion parameters. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks.

    “Studies demonstrate that pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks [23,150].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [150]

    Section 3.1.5 (pp. 13–14) compares a pre-trained, fine-tuned model with one trained on downstream tasks without pre-training and reports significant gains from pre-training on almost all benchmarks. It does not establish that pre-training by itself accounts for the bulk of downstream capabilities; the reported model is also fine-tuned, and gains are marginal on WMT English-to-French. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Even works that include significant curation efforts for the sake of improving models focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes.

    “Even works that do include significant curation efforts for the sake of improving models [57, 113] focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes [19].”

    From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [113]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Many datasets such as OpenWebText, RefinedWeb and C4 consist entirely of web-scraped information exposed to data poisoning.

    “Many datasets such as OpenWebText24, RefinedWeb25 and C4 (ref. 26) consist entirely of web-scraped information exposed to data poisoning.”

    From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [26]

    Section 2.2 (pp. 5–6) says C4 was assembled from Common Crawl’s web-extracted text, supporting the web-scraped description of C4, which is the dataset attached to this citation. The paper does not address C4’s exposure to data poisoning; the citations for OpenWebText and RefinedWeb are separate. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The selected pretraining corpora predominantly feature text from a broad spectrum of internet webpages, including Wikipedia, news sites and Reddit pages, and some include specialized domains such as arXiv, PubMed and StackExchange.

    “These diverse corpora, which have been widely used in training state-of-the-art LLMs, predominantly feature text from a broad spectrum of internet webpages, including sources such as Wikipedia, news sites and Reddit pages. Some of these corpora also include data from specialized domains, such as arXiv, PubMed and StackExchange.”

    From Generative language models exhibit social identity biases (2024) · cites this paper as [48]

    Section 2.2 (pp. 5–6) describes C4 as web-scraped text, and Section 3.4.1 (p. 26) describes Wikipedia, news-site and Reddit-filtered webpage corpora. The Reddit-filtered corpus contains webpages submitted to Reddit, not necessarily Reddit pages. The copy does not identify arXiv, PubMed or StackExchange as sources in the pretraining corpora (searched: those domain names). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • WNLI example places the pronoun in the hypothesis, not the premise

    Minor

    Appendix B says to locate 'they' in the premise, but its stated premise, 'The demonstrators feared violence,' contains no pronoun. 'They' appears in the sentence it labels the hypothesis.

  • Largest model is not best on every task

    Minor

    Table 14 reports an EnRo BLEU score of 28.2 for T5-3B and 28.1 for T5-11B, contrary to the claim that the 11B model performed best across all tasks.

  • Deshuffling example adds a word rather than just shuffling tokens

    Minor

    The deshuffled input in Table 3 contains 'fun', which is absent from the original sentence. A token shuffle cannot introduce that word.

  • Inverse-square-root learning rate is described as exponential decay

    Minor

    After warm-up, the specified learning rate varies as 1/√n, not exponentially with the training iteration.

  • Denoising does not always outperform language modeling

    Minor

    The stated conclusion has exceptions in Table 2. For the language-model architecture, the language-modeling objective scores 56.51 on SuperGLUE versus 55.02 for denoising, and 25.23 versus 25.09 on EnDe.

5 of 5

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
21
Issue
140
Pages
1-67
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.10683
arXiv
1910.10683
OpenAlex
W2981852735
Cited by
10,929
References
125
Citations by year
2026: 57 · 2025: 1758 · 2024: 2734 · 2023: 3457 · 2022: 1678 · 2021: 833 · 2020: 399 · 2019: 11 · 2013: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Journal of Machine Learning Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
transfer learning, text-to-text transformers, pre-training objectives, pre-trained language models, language understanding tasks, summarization, question answering, text classification, fine-tuning, unlabeled data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review