The DOI matches its DataCite record; 164 of the 164 references it cites were found in a registry or an index; 37 of the 57 sources its claims cite were checked for support.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Open accessAbstract
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces T5, a unified text-to-text framework that casts every NLP task as text-in/text-out, and uses it to systematically compare transfer-learning choices. Combining insights from that study with scale and the new C4 corpus yields state-of-the-art results on many benchmarks, including GLUE, SuperGLUE, SQuAD, and CNN/Daily Mail.
Key findings
- The text-to-text framework works across generative, classification, and regression tasks; T5-11B achieved a GLUE average of 90.3, SuperGLUE average of 88.9, SQuAD EM of 91.26, and CNN/Daily Mail ROUGE-2 of 21.55, reaching state of the art on 18 of 24 tasks considered.
- Denoising objectives generally outperformed language modeling and deshuffling; an encoder-decoder architecture performed best, though sharing encoder/decoder parameters performed nearly as well, and span corruption with mean span length 3 slightly outperformed i.i.d. corruption on most non-translation benchmarks.
- Scaling helped consistently: training longer, using larger models, and ensembling all improved performance; the 11B model performed best across tasks, and T5-Base substantially outperformed a baseline trained on the same 1T tokens, indicating non-scaling changes also mattered.
- C4 filtering mattered: unfiltered C4 performed worst in every task, while in-domain pre-training data sometimes helped specific downstream tasks; repeating a small pre-training data set many times degraded performance.
Methods
- The paper is primarily an empirical survey using a “coordinate ascent” approach: start from a baseline encoder-decoder Transformer and vary one factor at a time, including architecture, unsupervised objective, pre-training data, training strategy, and scaling.
- The baseline model has about 220M parameters, is pre-trained for 2^19 steps on C4, fine-tuned for 2^18 steps, and evaluated on GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT English-to-German/French/Romanian.
- C4 is built from April 2019 Common Crawl with heuristic cleaning, producing about 750 GB of English text; final T5 variants range from 60M to 11B parameters, are pre-trained on about 1T tokens with multi-task pre-training, and use beam search for translation and summarization.
Limitations
- The paper states that its coordinate-ascent approach may miss second-order effects, that it cannot cover every transfer-learning technique, and that its baseline cannot be directly compared to BERT because it is an encoder-decoder model pre-trained for roughly one-quarter as many steps.
- It reports high inter-run variance on low-resource tasks such as CoLA, CB, and COPA; notes it did not achieve state of the art on WMT translation, partly due to English-only unlabeled data and lack of backtranslation; and says SQuAD test-server resources were insufficient for its largest models.
- The text also makes evident that the work depends on very large-scale TPU computation and that English-only pre-training limits multilingual applicability; the paper itself notes that ROUGE improvements may not correspond to more coherent summaries and that some reading-comprehension metrics may be biased toward machine predictions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
89 Own result · 48 Own interpretation · 21 No basis given · 15 Partly supported · 12 Own method · 8 Supported · 5 Sources not found · 1 Supported so far · 1 Couldn’t confirm · 1 No full text of the sources · 1 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Omitting results on the WNLI validation set is standard practice (Devlin et al., 2018) due to the fact that it is 'adversarial' with respect to the training set, i.e. validation examples are all slightly-perturbed versions of training examples with the opposite label.”
What the sources say
- Partly supportedcites Devlin et al., 2018 sourceExperimental — accuracy not yet measured
Appendix B.1 (p. 15) says BERT excluded WNLI from its reported results, citing dataset-construction issues, poor performance by submitted systems, and fairness to OpenAI GPT. It does not establish that omission was standard practice or say that validation examples are perturbed training examples with opposite labels. (LLM, conf=0.87)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Finally, we confirm the widely-held conception that using a denoising objective always results in better downstream task performance compared to a language modeling objective.”
What the sources say
- Partly supportedcites Devlin et al., 2018 sourceExperimental — accuracy not yet measured
Section 5.1 (p. 8) reports that BERT’s masked-language-model variant outperformed its left-to-right language-model variant on every task tested. That supports the narrower comparison, but not the claim that a denoising objective always improves downstream performance: the paper evaluates particular models and tasks, and Section 3.1 (p. 4) calls its objective a masked language model rather than a denoising auto-encoder. (LLM, conf=0.88)
- Partly supportedcites Voita et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) proposes a hypothesis for why masked language modeling (MLM) may be preferable to language modeling (LM) for pretraining, and the Conclusions (p. 9) call it a possible explanation for MLM’s superior performance. The paper does not establish that a denoising objective always produces better downstream-task performance; the claim turns a qualified explanation into a universal result. (LLM, conf=0.88)
- Partly supportedcites Lample and Conneau, 2019 sourceExperimental — accuracy not yet measured
Section 5.3 (pp. 6–7) reports that masked language modeling (MLM) pretraining outperformed causal language modeling (CLM) pretraining in the machine-translation comparisons studied. It does not establish that a denoising objective always yields better performance across downstream tasks; the paper also distinguishes MLM pretraining from the denoising auto-encoding loss used during translation training. (LLM, conf=0.88)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For SQuAD, we outperformed the previous state-of-the-art (ALBERT (Lan et al., 2019)) by over one point on the Exact Match score.”
What the sources say
- Supportedcites Lan et al., 2019 source
The Introduction (p. 2) says ALBERT established a new state of the art on SQuAD, and §4.9 (p. 10) reports its SQuAD 2.0 result. The citation identifies ALBERT as the prior benchmark; the claim that T5 exceeded it by over one Exact Match point is the citing authors’ own comparison, not a result attributed to the ALBERT paper. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our baseline model uses a relatively small amount of pre-training ( 1 / 4 as much as BERT (Devlin et al., 2018), 1 / 16 as much as XLNet (Yang et al., 2019), 1 / 64 as much as RoBERTa (Liu et al., 2019c), etc.).”
What the sources say
- Partly supportedcites Devlin et al., 2018 sourceExperimental — accuracy not yet measured
Appendix A.2 (p. 13) reports BERT’s pre-training amount: 1,000,000 steps at 128,000 tokens per batch. The paper does not describe the citing authors’ baseline, so it cannot establish that the baseline used one-quarter as much pre-training as BERT. The XLNet and RoBERTa comparisons are attributed to other citations. (LLM, conf=0.86)
- Partly supportedcites Yang et al., 2019 sourceExperimental — accuracy not yet measured
Section 3.1 (p. 6) reports that XLNet-Large was pretrained for 500,000 steps with a batch size of 8,192, supporting the cited figure as an XLNet pretraining benchmark. The paper does not describe the citing authors’ baseline, so it cannot establish that the baseline used 1/16 as much pretraining as XLNet. (LLM, conf=0.83)
- Not matchedcites Liu et al., 2019c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While many modern approaches to transfer learning for NLP use a Transformer architecture consisting of only a single 'stack' (e.g. for language modeling (Radford et al., 2018; Dong et al., 2019) or classification and span prediction (Devlin et al., 2018; Yang et al., 2019)), we found that using a standard encoder-decoder structure achieved good results on both generative and classification tasks.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also explore the limits of current approaches by scaling up the insights from our systematic study (training models up to 11 billion parameters) to obtain state-of-the-art results in many of the tasks we consider.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The WNLI training and validation sets have a significant overlap with the WSC training set.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that most of these scores were annotated in increments of 0 . 2, so we simply rounded any score to the nearest increment of 0 . 2 and converted the result to a literal string representation of the number”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all tasks, the encoder-decoder architecture with the denoising objective performed best.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 54
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, we opted to create a new data set because prior data sets use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope (e.g. are limited to News data (Zellers et al., 2019; Liu et al., 2019c), comprise only Creative Commons content (Habernal et al., 2016), or are focused on parallel training data for machine translation (Smith et al., 2013)).”
What the sources say
- Supportedcites Zellers et al., 2019 source
The dataset description in Section 3 (p. 4) identifies RealNews as a corpus of news articles and says its collection was limited to domains indexed by Google News. This supports the news-only scope for which the citing sentence invokes Zellers et al.; the other datasets and reasons listed in that sentence are not attributed to this source. (LLM, conf=0.93)
- Not matchedcites Liu et al., 2019c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Habernal et al., 2016 source
The Habernal et al. citation is attached specifically to the example that the dataset comprises Creative Commons content, not to every reason given for creating a new dataset. The Introduction identifies C4Corpus as the “Creative Commons from Common Crawl Corpus,” and the Results describe its pages by Creative Commons license. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We also experimented with using a triangular learning rate (Howard and Ruder, 2018), which produced slightly better results but requires knowing the total number of training steps ahead of time.”
What the sources say
- Supportedcites Howard and Ruder, 2018 source
The citation identifies the learning-rate schedule the citing authors used, not the result of their own comparison with an inverse-square-root schedule. In “Discriminative fine-tuning,” Howard and Ruder describe slanted triangular learning rates and define their schedule using T, the total number of training iterations. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings (Radford et al., 2018; Devlin et al., 2018; McCann et al., 2018; Yu et al., 2018).”
What the sources say
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Devlin et al., 2018 source
Devlin et al. provide an example of the Transformer’s subsequent use across NLP settings. The Introduction (pp. 1–2) describes BERT as a Transformer-based model and reports state-of-the-art results on eleven NLP tasks, including sentence-level and token-level tasks. (LLM, conf=0.91)
- Partly supportedcites McCann et al., 2018 sourceExperimental — accuracy not yet measured
Further Related Work (p. 17) says attention-only models were effective for machine translation. The MQAN section (pp. 5–6) describes using Transformer-derived self-attention layers across decaNLP tasks, but the model also uses recurrent and coattentive layers; it does not establish that the Transformer architecture itself was used across those settings. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Corrupting spans was also previously considered as a pre-training objective for BERT, where it was found to improve performance (Joshi et al., 2019).”
What the sources say
- Supportedcites Joshi et al., 2019 source
Section 6.1 (p. 8) tests span masking as a replacement for BERT’s token-masking scheme. Table 6 (p. 8) shows improved performance on several downstream tasks, though coreference performance is lower; the claim does not say improvement was universal. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Preliminary work that applied the transfer learning paradigm of pre-training and fine-tuning all of the model's parameters to NLP problems used a causal language modeling objective for pre-training (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018).”
What the sources say
- Supportedcites Dai and Le, 2015 source
The Introduction (p. 1) describes next-step prediction with a recurrent language model as an unsupervised pretraining method for NLP classification. The Experiments section (p. 3) says the pretrained embeddings and LSTM weights initialize the supervised model and are both fine-tuned. The paper also studies a sequence-autoencoder objective, but that does not negate its use of causal language-model pretraining. (LLM, conf=0.94)
- Partly supportedcites Peters et al., 2018 sourceExperimental — accuracy not yet measured
Section 3.1 describes pre-training with forward and backward language models that predict tokens from preceding or following context, respectively. But Section 3.3 says the pre-trained biLM weights are frozen when ELMo is added to supervised task models. Peters et al. therefore support the language-modeling objective, not the attributed fine-tuning of all model parameters. (LLM, conf=0.90)
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is considerably less than BERT (Devlin et al., 2018), which used roughly 137B tokens, or RoBERTa (Liu et al., 2019c), which used roughly 2 . 2T tokens.”
What the sources say
- Supportedcites Devlin et al., 2018 source
The BERT citation supports the BERT portion of the sentence; RoBERTa has a separate citation. Appendix A.2 (p. 13) reports a nominal 128,000 tokens per batch for 1,000,000 steps, yielding about 128B tokens—close to the cited rough figure of 137B. It also says 90% of steps used shorter sequences, so the paper does not establish 137B as a precise count of tokens processed. (LLM, conf=0.82)
- Not matchedcites Liu et al., 2019c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, it has recently been shown that 'denoising' objectives (Devlin et al., 2018; Taylor, 1953) (also called 'masked language modeling') produce better performance and as a result they have quickly become standard.”
What the sources say
- Partly supportedcites Devlin et al., 2018 sourceExperimental — accuracy not yet measured
Section 4.1 (p. 6) reports that BERT outperformed the left-to-right OpenAI GPT model on the GLUE tasks, supporting the performance comparison. Section 3.1 (p. 4) identifies BERT’s masked-language-model objective, but the paper does not establish that such objectives had become standard; it also notes that BERT’s training differs from GPT’s in other ways (Appendix A.4, p. 14). (LLM, conf=0.88)
- Not checked — no full text of the sourcecites Taylor, 1953 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al., 2018; Yang et al., 2019; Dong et al., 2019; Liu et al., 2019c; Lan et al., 2019).”
What the sources say
- Supportedcites Devlin et al., 2018 source
The Introduction (p. 1) says language-model pre-training improves many NLP tasks. Section 4.1 (p. 6) reports that pre-trained BERT surpassed prior state-of-the-art systems across the GLUE tasks, supporting the claim that this approach achieved state-of-the-art benchmark results. (LLM, conf=0.91)
- Supportedcites Yang et al., 2019 source
The Introduction (p. 1) describes pretraining on large unlabeled text corpora, and the paper evaluates XLNet across language-understanding tasks. The results in Section 3.3 (pp. 7–8) compare XLNet with state-of-the-art results on RACE, GLUE, and several text-classification benchmarks, supporting the claim for this cited example. (LLM, conf=0.91)
- Supportedcites Dong et al., 2019 source
The Introduction (p. 1) says language-model pre-training has advanced the state of the art across a variety of NLP tasks and describes UNILM's unsupervised pre-training. The Introduction (p. 2) reports new state-of-the-art results on five NLG datasets, supporting the citing sentence's broad claim about benchmark results. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Unlabeled text data is available en masse thanks to the Internet.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
What the sources say
- Supportedcites Beltagy et al. (2019) source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The 'bitter lesson' of machine learning research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise (Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b; Keskar et al., 2019a).”
What the sources say
- Not matchedcites Sutton, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Hestness et al., 2017 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes scaling data and computation as a lower-risk complement to creative model-architecture search, and the Conclusion (p. 13) says the resulting accuracy improvements are empirically predictable. The paper does not claim that computation-leveraging methods ultimately win over methods relying on human expertise. (LLM, conf=0.86)
- Partly supportedcites Shazeer et al., 2017 sourceExperimental — accuracy not yet measured
Section 1.1 (p. 1) says that increasing neural-network capacity can improve prediction accuracy, and the Abstract (p. 1) reports improved benchmark results from a much larger model. The paper does not compare general methods with methods relying on human expertise or claim that the former ultimately win out. (LLM, conf=0.82)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“As pointed out by Arivazhagan et al. (2019), an extremely important factor in multi-task learning is how much data from each task the model should be trained on.”
What the sources say
- Supportedcites Arivazhagan et al. (2019) source
Section 4.2 (pp. 7–8) examines how the proportion of training samples drawn from each language-pair task affects model performance, finding that a balanced sampling strategy changes the trade-off between transfer and interference. Section 8 (p. 17) says that even simple data-sampling and balancing heuristics can significantly improve outcomes for individual tasks. This supports the citing sentence’s point, though the paper demonstrates it in multilingual neural machine translation. (LLM, conf=0.91)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“though it has been shown that the multi-task setup can confer benefits across very similar tasks Liu et al. (2019b); Ratner et al. (2018).”
What the sources say
- Not matchedcites Liu et al. (2019b)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Ratner et al. (2018) source
The Introduction describes the studied tasks as hierarchically related decisions within a fine-grained classification problem. The Experiments section’s End Model Ablation (page unavailable) reports the best performance on each task with the hierarchical multi-task configuration, supporting the qualified claim that a multi-task setup can benefit closely related tasks. (LLM, conf=0.88)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Pursuant with past results (Houlsby et al., 2019; Bapna et al., 2019) we find that lower-resource tasks like SQuAD work well with a small value of d whereas higher resource tasks require a large dimensionality to achieve reasonable performance.”
What the sources say
- Partly supportedcites Houlsby et al., 2019 sourceExperimental — accuracy not yet measured
Section 3.5 (p. 6) reports that SQuAD performs well with adapters as small as size 2. But the paper does not identify SQuAD as a lower-resource task or establish that higher-resource tasks require a larger dimension. Section 3.6 (p. 7) instead reports stable performance across adapter sizes 8, 64, and 256 on its classification tasks. (LLM, conf=0.88)
- Not matchedcites Bapna et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Baevski et al. (2019) similarly observed that truncating the pre-training data set size can degrade downstream task performance.”
What the sources say
- Supportedcites Baevski et al. (2019) source
Section 6.4, “Domain and amount of training data” (page not provided), reports that more pretraining data significantly increases end-task accuracy and concludes that more data improves performance when other factors are held equal. This supports the claim that reducing the pretraining dataset can degrade downstream performance. (LLM, conf=0.93)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This is the method used by the 'MT-DNN' (Liu et al., 2015, 2019b), which achieved state-of-the-art performance on GLUE and other benchmarks when it was introduced.”
What the sources say
- Could not verifycites Liu et al., 2015 sourceExperimental — accuracy not yet measured
The available passages are bibliographic metadata for the 2015 paper (pp. 912–921), not its results. Searches for GLUE, MT-DNN, and state-of-the-art benchmark performance returned no passage addressing the claim, so our copy cannot verify it. (LLM, conf=0.75)
- Not matchedcites Liu et al., 2019b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent results suggest that this may hold true for transfer learning in NLP (Liu et al., 2019c; Radford et al., 2019; Yang et al., 2019; Lan et al., 2019), i.e. it has repeatedly been shown that scaling up produces improved performance compared to more carefully-engineered methods.”
What the sources say
- Not matchedcites Liu et al., 2019c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Radford et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Yang et al., 2019 sourceExperimental — accuracy not yet measured
Section 3.1 (p. 6) says XLNet scaled up training, and Section 3.3 (p. 7) reports that it generally outperformed BERT and RoBERTa. But the paper also attributes gains to XLNet’s novel pretraining objective and architectural changes (Introduction, p. 2); it does not show that scaling up, rather than those engineered changes, produced better performance than more carefully engineered methods. (LLM, conf=0.84)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The finding that a multi-task model is outperformed by separate models trained on each individual task has previously been observed e.g. by Arivazhagan et al. (2019) and McCann et al. (2018)”
What the sources say
- Partly supportedcites Arivazhagan et al. (2019) sourceExperimental — accuracy not yet measured
Section 4.1 (p. 6) reports that multilingual models perform worse than separate bilingual baselines on high-resource language pairs. But it also reports that joint training beats those baselines on low-resource pairs, so the paper does not support an unqualified claim that separate models outperform the multi-task model on each task. (LLM, conf=0.88)
- Partly supportedcites McCann et al. (2018) sourceExperimental — accuracy not yet measured
Appendix D (pp. 22–23) reports that fully joint training struggles to reach single-task performance on several tasks and that a gap remains. But the Introduction (p. 2) says the multitask MQAN, with the right training strategy, achieves performance comparable to ten separately trained MQANs; Related Work (p. 9) says it performs nearly as well or better on each task. The claim overstates the paper’s finding as a general result that separate models outperform the multitask model. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It has been argued that fine-tuning all of the model's parameters can lead to suboptimal results, particularly on low-resource tasks (Peters et al., 2019).”
What the sources say
- Partly supportedcites Peters et al., 2019 sourceExperimental — accuracy not yet measured
The Conclusion (page unavailable) says that fine-tuning can perform differently from feature extraction depending on how similar the pretraining and target tasks are; the Impact of additional parameters section also describes fine-tuning underperforming in dissimilar settings. The cited paper does not identify low-resource tasks as a particular reason that tuning all parameters yields suboptimal results. (LLM, conf=0.84)
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, training with a 4 × larger batch size can yield a different outcome than training for 4 × as many steps (Shallue et al., 2018).”
What the sources say
- Supportedcites Shallue et al., 2018 source
In §1.2, the paper reports that increasing batch size initially reduces the steps needed to reach a target error proportionally, but that the benefit diminishes and eventually disappears. This supports the claim that a fourfold increase in batch size need not produce the same outcome as four times as many training steps; the claim does not assert that the outcomes always differ. (LLM, conf=0.86)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Early results on transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model (Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017).”
What the sources say
- Supportedcites Subramanian et al., 2018 source
The Evaluation Strategy (p. 6) says the authors use learned sentence representations as features for a low-complexity classifier on new supervised tasks, without updating the sentence-representation model. The Experimental Results (p. 6) report training logistic regression on those fixed representations, matching the claimed approach. (LLM, conf=0.94)
- Supportedcites Kiros et al., 2015 source
The Experiments section (p. 4) says the authors extracted vectors from a pretrained encoder and trained a linear classifier with no fine-tuning or backpropagation through the encoder. The Introduction (p. 2) likewise describes freezing the model and training linear models for tasks that include classification benchmarks. (LLM, conf=0.89)
- Not checked — no full text of the sourcecites Logeswaran and Lee, 2018 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Some recent notable exceptions (Baevski et al., 2019; Liu et al., 2019c; Yang et al., 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).”
What the sources say
- Supportedcites Baevski et al., 2019 source
Section 6.4 compares pretraining on up to 18B Common Crawl tokens with pretraining on BooksCorpus and English Wikipedia, and Table 6 reports results across these data sources. Section 5.1 identifies BooksCorpus plus Wikipedia as the smaller dataset similar to BERT’s training data. (LLM, conf=0.93)
- Not matchedcites Liu et al., 2019c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Yang et al., 2019 source
Section 3.1 (p. 6) says the authors trained XLNet-Large on the smaller BooksCorpus-and-Wikipedia collection and then scaled it up using additional data, including 110 GB of Common Crawl text. This supports the cited sentence’s description of contrasting smaller preexisting pretraining data with a much larger collection. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Unfortunately, the majority of the resulting text is not natural language. Instead, it largely comprises gibberish or boiler-plate text like menus, error messages, or duplicate text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that the choice of text prefix used for a given task is essentially a hyperparameter; we found that changing the exact wording of the prefix had limited impact and so did not perform extensive experiments into different prefix choices.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Common Crawl produces about 20TB of text data extracted from web pages each month.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, we always count the model's output as wrong, though we never observed this behavior in any of our trained models.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is WMT English to French, which is a large enough data set that gains from pre-training tend to be marginal.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Drop corrupted tokens | 84 . 44 | 19 . 31 | 80 . 52 | 68 . 67 | 27 . 07 | 39 . 76 | 27 . 82”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, however, we observed nearly identical step times for L -layer language models versus L + L -layer encoder-decoder models, suggesting a roughly equivalent computational cost.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No pre-training | 66 . 22 | 17 . 60 | 50 . 31 | 53 . 04 | 25 . 86 | 39 . 77 | 24 . 04”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Drop corrupted tokens | 84 . 44 | 19 . 31 | 80 . 52 | 68 . 67 | 27 . 07 | 39 . 76 | 27 . 82”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, we find a limited difference between these objectives, though the version with an average span length of 10 slightly underperforms the other values in some cases.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“⋆ Replace corrupted spans | 83 . 28 | 19 . 24 | 80 . 88 | 71 . 36 | 26 . 98 | 39 . 82 | 27 . 65”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BERT BASE achieved an exact match score of 80 . 8 on SQuAD and an accuracy of 84 . 4 on MNLI-matched, whereas we achieve 80 . 88 and 84 . 24, respectively”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“⋆ Baseline average | 83 . 28 | 19 . 24 | 80 . 88 | 71 . 36 | 26 . 98 | 39 . 82 | 27 . 65”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that 2 35 tokens only covers a fraction of the entire C4 data set, so we never repeat any data during pre-training.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Baseline standard deviation | 0 . 235 | 0 . 065 | 0 . 343 | 0 . 416 | 0 . 112 | 0 . 090 | 0 . 108”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also note that the shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This variant has the highest parameter count (2 P ) but the same computational cost as the P -parameter decoder-only models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, using the Wikipedia + TBC corpus produced a SuperGLUE score of 73 . 24, beating our baseline's score (using C4) of 71 . 36.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that gradual unfreezing caused a minor degradation in performance across all tasks, though it did provide some speedup during fine-tuning.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, we note that temperature-scaled mixing also provides a means of obtaining reasonable performance from most tasks, with T = 2 performing the best in most cases.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A first obvious takeaway is that removing the heuristic filtering from C4 uniformly degrades performance and makes the unfiltered variant perform the worst in every task.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We combined our above insights and trained substantially larger models (up to 11 billion parameters) to achieve state-of-the-art results across many of the benchmarks we considered.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found each approach conferred a significant boost in performance, though training a smaller model on more data was often outperformed by training a larger model for fewer steps.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In spite of its simplicity, we found the text-totext framework obtained comparable performance to task-specific architectures and ultimately produced state-of-the-art results when combined with scale.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no clear winner between training for 4 × as many steps or using a 4 × larger batch size, though both were beneficial.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also showed that sharing the parameters in the encoder and decoder did not result in a substantial performance drop while halving the total parameter count.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An unsurprising but important result from our study is that larger models tend to perform better.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the basic approach of updating all of a pre-trained model's parameters during fine-tuning outperformed methods that are designed to update fewer parameters, although updating all parameters is most expensive.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsurprisingly, increasing the training time and/or model size consistently improves the baseline.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In terms of parameter count, our 11B model variant is the largest model that has been submitted to the GLUE benchmark.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We showed how this approach can be successfully applied to generative tasks like abstractive summarization, classification tasks like natural language inference, and even regression tasks like STS-B.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite these potential issues, we find that our models do generate coherent and largely correct summaries.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To show that our model is generating fluent summaries, we include a few example decodes from our best model (T5-11B) on the validation set along with the ground-truth summaries.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, on CNN/Daily Mail we attain state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To avoid leaking validation examples into our training data (a particular issue in the multi-task experiments of Section 3.5.2), we therefore never train on WNLI and never report results on the WNLI validation set.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, we do not include WNLI in the average GLUE score whenever we report on the validation set (all sections except Section 3.7 where results are presented on the test sets).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all of the experiments except those in Section 3.7, we report results in the validation set to avoid performing model selection on the test set.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using a larger corruption rate also results in longer targets, which can potentially slow down training.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To see why, first note an encoder-decoder model with L layers in the encoder and L layers in the decoder has approximately the same number of parameters as a language model with 2 L layers. However, the same L + L encoder-decoder model will have approximately the same computational cost as a language model with only L layers.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As a cheaper alternative, we train our baseline model 10 times from scratch (i.e. with different random initializations and data set shuffling)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, our framework also allows for generative tasks like machine translation and abstractive summarization where it is not possible to enumerate all possible output choices.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two variants that do not require predicting the full original sequence ('replace corrupted spans' and 'drop corrupted spans') are both potentially attractive since they make the target sequences shorter and consequently make training faster.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that we cannot directly compare our baseline to BERT BASE because ours is an encoder-decoder model and was pre-trained for roughly 1 / 4 as many steps.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We save a checkpoint every 5 , 000 steps and report results on the model checkpoint corresponding to the highest validation performance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This value was chosen as a trade-off between the high-resource tasks (i.e. those with large data sets), which benefit from additional fine-tuning, and low-resource tasks (smaller data sets), which overfit quickly.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using only 2 35 tokens results in a reasonable computational budget while still providing a sufficient amount of pre-training for acceptable performance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For models fine-tuned on multiple tasks, we choose the best checkpoint for each task independently.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our choices to mask consecutive spans of tokens and only predict dropped-out tokens were made to reduce the computational cost of pre-training.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This 'coordinate ascent' approach might miss second-order effects (for example, some particular unsupervised objective may work best on a model larger than our baseline setting), but performing a combinatorial exploration of all of the factors in our study would be prohibitively expensive.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In contrast, modern techniques for transfer learning in NLP often pre-train using unsupervised learning on unlabeled data.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Unlabeled text data is available en masse thanks to the Internet.
In the paper
“In contrast, most applications of multi-task learning to NLP add task-specific classification networks or use different loss functions for each task (Liu et al., 2019b).”
What the sources say
- Not matchedcites Liu et al., 2019b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could suggest that the translation tasks benefit less from (English) pre-training, whereas unsupervised pre-training is an important factor in the other tasks.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fortunately, the span-corruption objective also provides some speedup during training compared to the i.i.d. noise approach because span corruption produces shorter sequences on average.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also note that this effect may be more pronounced for larger model sizes, i.e. a bigger model may be more prone to overfitting to a smaller pre-training data set.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“A similar exploration was performed by Wang et al. (2019a).”
What the sources say
- Not matchedcites Wang et al. (2019a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, different objectives (or parameterizations of objectives) can lead to different sequence lengths and thus different training speeds.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Liu et al. (2019c) also observed that pre-training on a more diverse data set yielded improvements on downstream tasks.”
What the sources say
- Not matchedcites Liu et al. (2019c)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This motivates the use of a large and diverse data set like C4 for generic language understanding tasks.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though an encoder-decoder model uses twice as many parameters as 'encoder-only' (e.g. BERT) or 'decoder-only' (language model) architectures, it has a similar computational cost.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that as discussed in Section 2.4, we do not train on any of the data from WNLI.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set, so we can still compare to what is ostensibly the state-of-the-art.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The model with the highest performance on the SQuAD test set also reported results on the validation set.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, the computational resources on this server are insufficient for obtaining predictions from our largest models. As a result, we instead continue to report performance on the SQuAD validation set.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The computational resources on the SQuAD test server are insufficient for obtaining predictions from the largest models.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results stem from the combination of a straightforward and unified text-to-text framework, our new C4 data set, and insights from our systematic study.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These examples selected at random and were not cherry-picked.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
Common Crawl produces about 20TB of text data extracted from web pages each month.
“the Common Crawl project 2 produces about 20TB of text data extracted from web pages each month.”
The paper relies on this without support.
The architectural changes (removing Layer Norm bias, placing layer normalization outside the residual path, and using a different position embedding scheme) are orthogonal to the experimental factors considered in the empirical survey of transfer learning.
“Since these architectural changes are orthogonal to the experimental factors we consider in our empirical survey of transfer learning, we leave the ablation of their impact for future work.”
The paper relies on this without support.
Unlabeled text data is available en masse thanks to the Internet.
“Beyond its empirical strength, unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet”
The authors state this as an assumption.
The variance over the 10 baseline runs also applies to each experimental variant.
“we train our baseline model 10 times from scratch (i.e. with different random initializations and data set shuffling) and assume that the variance over these runs of the base model also applies to each experimental variant.”
The paper relies on this without support.
Most changes made in the experiments do not have a dramatic effect on the inter-run variance.
“We don't expect most of the changes we make to have a dramatic effect on the inter-run variance, so this should provide a reasonable indication of the significance of different changes.”
The authors state this as an assumption.
An L+L-layer encoder-decoder model has the same number of parameters as a 2L-layer language model.
“we make the simplifying assumption that an L + L -layer encoder-decoder model has the same number of parameters as an 2 L -layer language model.”
The paper relies on this without support.
New pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code.
“However, unlike objectives and benchmarks, new pre-training data sets are usually not treated as significant contributions on their own and are often not released alongside pre-trained models and code.”
The paper relies on this without support.
There has been relatively little comparison of different pre-training data sets and a lack of a standard data set used for pre-training.
“As a result, there has been relatively little comparison of different pre-training data sets as well as a lack of a 'standard' data set used for pre-training.”
The paper relies on this without support.
MultiRC's largest source of data comes from fiction books, which is exactly the domain covered by TBC.
“MultiRC is a reading comprehension data set whose largest source of data comes from fiction books, which is exactly the domain covered by TBC.”
The paper relies on this without support.
ReCoRD is a data set that measures reading comprehension on news articles.
“ReCoRD, a data set that measures reading comprehension on news articles.”
The paper relies on this without support.
SQuAD is a question-answering data set with passages sourced from Wikipedia.
“SQuAD, which is a question-answering data set with passages sourced from Wikipedia.”
The paper relies on this without support.
Obtaining additional unlabeled data is cheap and easy.
“obtaining additional unlabeled data is cheap and easy”
The paper relies on this without support.
There is potential for task interference or negative transfer, where achieving good performance on one task can hinder performance on another.
“An additional issue is the potential for 'task interference' or 'negative transfer', where achieving good performance on one task can hinder performance on another.”
The paper relies on this without support.
The pre-training objective is stochastic and can help prevent the model from seeing the same exact data multiple times.
“because our pre-training objective is itself stochastic and can help prevent the model from seeing the same exact data multiple times.”
The paper relies on this without support.
Leave-one-out multi-task training simulates the real-world setting where a pre-trained model is fine-tuned on a task it had not seen during pre-training.
“This simulates the real-world setting where a pre-trained model is fine-tuned on a task it had not seen during pre-training.”
The paper relies on this without support.
Multi-task pre-training provides a diverse mixture of supervised tasks.
“Note that multi-task pre-training provides a diverse mixture of supervised tasks.”
The authors state this as an assumption.
Larger models trained for longer might benefit from a larger proportion of unlabeled data because they are more likely to overfit to smaller training data sets.
“We hypothesize that larger models trained for longer might benefit from a larger proportion of unlabeled data because they are more likely to overfit to smaller training data sets.”
The paper relies on this without support.
The computational resources on the SQuAD test server are insufficient for obtaining predictions from the largest models.
“Unfortunately, the computational resources on this server are insufficient for obtaining predictions from our largest models.”
The paper relies on this without support.
The model with the highest performance on the SQuAD test set also reported results on the validation set.
“Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set”
The paper relies on this without support.
Humans achieve 100% accuracy on both COPA and WSC.
“humans achieve 100% accuracy on both COPA and WSC”
The paper relies on this without support.
C4 is so large that the authors do not complete one pass over the data even when training for 2^23 steps.
“C4 is so large that we do not complete one pass over the data even when training for 2 23 steps.”
The paper relies on this without support.
The primary concern in multi-task learning is setting the proportion of each task to train on.
“The primary concern in multi-task learning is setting the proportion of each task to train on.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs refer to the recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance [9,16].”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies demonstrate that pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks [23,150].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [150]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even works that do include significant curation efforts for the sake of improving models [57, 113] focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes [19].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [113]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many datasets such as OpenWebText24, RefinedWeb25 and C4 (ref. 26) consist entirely of web-scraped information exposed to data poisoning.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These diverse corpora, which have been widely used in training state-of-the-art LLMs, predominantly feature text from a broad spectrum of internet webpages, including sources such as Wikipedia, news sites and Reddit pages. Some of these corpora also include data from specialized domains, such as arXiv, PubMed and StackExchange.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [48]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They can do so without any access to an external memory, as a parameterized implicit knowledge base [51, 52].”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [51]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Based on an analysis of the C4 corpus, we estimate that alphabetical order is 150 times more common than reverse alphabetical order in Internet text.”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“Based on an analysis of the C4 corpus (36), we found that the three most common shift levels are rot-1, rot-3, and rot-13”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“C4, the dataset that the T5 language model was trained on [113], went unreleased for almost two years until it was replicated and shared by the Allen AI institute, enabling other scholars to study it and use it for training their own models.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [113]
The cited source says what the claim attributes to it.
In the citing paper
“This means that our conclusions, especially regarding consciousness, apply to other models such as BERT [1], GPT-3 [2], T5 [4] and future models using the same, or similar, architectures.”
From Deanthropomorphising NLP: Can a language model be conscious? (2024) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“T5 demonstrates that by scaling to very large models sizes, denoising sequence-to-sequence pre-training can achieve new state-of-the-art results on many tasks.”
From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Raffel et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“To juxtapose social identity biases measured in LLMs against human-level biases, we obtained human-written ingroup and outgroup sentences from large-scale web corpora commonly used to pretrain LLMs, including C448, The Pile55, OpenWebText56 and the November–December 2022 edition of OLM57.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [48]
The cited source says what the claim attributes to it.
In the citing paper
“LaMDA models use T5 position biases [16] to handle position information.”
From Exploring Length Generalization in Large Language Models (2022) · cites this paper as [16]
The cited source says what the claim attributes to it.
In the citing paper
“Like T5 [51] or BART, RAG can be fine-tuned on any seq2seq task, whereby both the generator and retriever are jointly learned.”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [51]
The cited source says what the claim attributes to it.
In the citing paper
“BART [32] and T5 [51, 52] propose a single, pre-trained encoder-decoder model that leverages bi-directional attention to achieve stronger performance on discriminative and generative tasks.”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [51]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To address this, entropy regularization techniques are introduced [71,93] to amplify diversity in the action space, albeit not always resolving the issue [151].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [151]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“PANN is a CNN-based [19] audio classification model with 7 downsampling CNN blocks and 7 upsampling blocks.”
From Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2023) · cites this paper as [19]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For the candidate test dataset, we further remove duplicates using CLIP [35] similarity and conduct a manual inspection on each entry.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [35]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Compared to the two fine-tuned baselines, Design2Code-18B [45] and WebSight VLM-8B [27], our model, fine-tuned from the smaller Pix2Struct-1.3B,outperforms both across all evaluation metrics, including CLIP-based visual similarity [35], low-level appearance accuracy [45], and our proposed TreeBLEU to measure the structural hierarchy recall.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Appendix B. Converting WNLI to Our Text-to-Text Format · p. 45
“the hypothesis would be 'The city councilmen refused the demonstrators a permit because they feared violence.' with the premise 'The demonstrators feared violence.' and the label False.”
§ Appendix B. Converting WNLI to Our Text-to-Text Format · p. 45
“We first find the location of all pronouns in the premise ('they' in our example).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.7 Putting It All Together · p. 38
“As expected, our largest (11 billion parameter) model performed best among our model size variants across all tasks.”
§ 3.7 Putting It All Together · p. 39
“| T5-3B | 31 . 8 | 42 . 6 | 42 . 6 | 28 . 2 | 42 . 72 | 21 . 02 | 39 . 94 | | T5-11B | 32 . 1 | 43 . 4 | 43 . 4 | 28 . 1 | 43 . 52 | 21 . 55 | 40 . 69 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.3 Unsupervised Objectives · p. 21
“| Deshuffling | party me for your to . last fun you inviting week Thank | (original text) |”
§ 3.3.1 Disparate High-Level Approaches · p. 21
“This approach takes a sequence of tokens, shuffles it, and then uses the original deshuffled sequence as a target.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1.2 Training · p. 12
“During pre-training, we use an 'inverse square root' learning rate schedule: 1 /√ max( n, k ) where n is the current training iteration and k is the number of warm-up steps (set to 10 4 in all of our experiments).”
§ 3.1.2 Training · p. 12
“then exponentially decays the learning rate until pre-training is over.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2.4 Results · p. 19
“using a denoising objective always results in better downstream task performance compared to a language modeling objective.”
§ 3.2.4 Results · p. 20
“| Language model | Denoising | P | M | 74 . 70 | 17 . 93 | 61 . 14 | 55 . 02 | 25 . 09 | 35 . 28 | 25 . 86 | | Prefix LM | Denoising | P | M | 81 . 82 | 18 . 61 | 78 . 94 | 68 . 11 | 26 . 43 | 37 . 98 | 27 . 39 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Self-citationImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Self-citationImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Cited by
Sorted by citations
Items 10,001–10,100 of 11,048
Statistical Depth for Ranking and Characterizing Transformer-Based Text Embeddings
Seegmiller, P., & Preum, S. M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.596
Scalar Anaphora: Annotating Degrees of Coreference in Text
Ye, B., Tu, J., & Pustejovsky, J. D. (2023) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2023.crac-main.4
Findings of the 1st Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2023
Tinner, F., Adelani, D. I., Emezue, C. C., Hajili, M., Goldman, O., Adilazuarda, M. F., Kautsar, M. D. A., Mirsaidova, A., Kural, M., Klenner, D. M. M., et al. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.24
Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue
Yang, Y., Huang, H., Liu, Y., & Gao, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.982
Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval
Jiang, F., Xu, Q., Drummond, T. W., & Cohn, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.65 · Cited by 3
Singhal, B., Gopalan, S., Krishna, A., & Chetlur, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.42
Zacharopoulos, C.-N., Desbordes, T., & Sablé-Meyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.998
Müller, B., Alastruey, B., Hansanti, P., Kalbassi, E., Ropers, C., Smith, E., Williams, A., Zettlemoyer, L., Andrews, P., & Costa‐jussà, M. R. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.48 · Cited by 2
Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis
Lu, J., Huang, W., Zheng, N., Zeng, X., Yeung, Y. T., & Chen, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.327 · Cited by 2
The student becomes the master: Outperforming GPT3 on Scientific Factual Error Correction
Ashok, D., Kulkarni, A., Pham, H., & Póczos, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.451
Test-time Augmentation for Factual Probing
Kamoda, G., Heinzerling, B., Sakaguchi, K., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.236 · Cited by 1
Strong and Efficient Baselines for Open Domain Conversational Question Answering
Coman, A. C., Barlacchi, G., & de Gispert, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.417
Smart “Chef”: Verifying the Effect of Role-based Paraphrasing for Aspect Term Extraction
Chen, J., Yu, H. Z., Xu, Q., & Yao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.144
Measuring bias in Instruction-Following models with P-AT
Onorati, D., Ruzzetti, E. S., Venditti, D., Ranaldi, L., & Zanzotto, F. M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.539
Few-shot Unified Question Answering: Tuning Models or Prompts?
Bansal, S., Yavuz, S., Pang, B., Bhat, M., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.550 · Cited by 3
Self-citationMore Than Spoken Words: Nonverbal Message Extraction and Generation
Yu, D., Wang, X., Chen, W., Du, N., Wang, L., Mi, H., & Yu, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1021
End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems
Towle, B., & Zhou, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.510
MUX-PLMs: Data Multiplexing for High-throughput Language Models
Murahari, V., Deshpande, A., Jimenez, C., Izhak Shafran, J., Wang, M., Cao, Y., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.301 · Cited by 6
Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering
Su, X., Howard, P., Hakim, N. Z., & Bethard, S. J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.67 · Cited by 2
Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning
Che, C., Yang, C. W., Lin, C.-Y., & Kao, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.110
Xue, H., & Αλέτρας, Ν. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.695
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Chen, W., Xu, X., Han, X., Lin, Y., Xie, R., Liu, Z., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.738
EpiK-Eval: Evaluation for Language Models as Epistemic Models
Prato, G., Huang, J., Parthasarathi, P., Sodhani, S., & Chandar, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.593 · Cited by 3
Natural Language Annotations for Reasoning about Program Semantics
Zocca, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.601
Compositional Generalization for Data-to-Text Generation
Xu, X., Titov, I., & Lapata, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.623 · Cited by 1
PARROT: Zero-Shot Narrative Reading Comprehension via Parallel Reading
Zhao, C., Vijjini, A. R., & Chaturvedi, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.895
Towards Concept-Aware Large Language Models
Shani, C., Vreeken, J., & Shahaf, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.877 · Cited by 2
Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading
Luo, Y., Tian, S., Yuan, C., & Wang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.866
Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation
Yang, C., Zheng, L., Wang, L., Tian, C., Pang, L., Li, J., Ho, Q., Cao, Y., & Wang, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.497 · Cited by 1
On the utility of enhancing BERT syntactic bias with Token Reordering Pretraining
Mesbahi, Y. E., Mahmud, A., Ghaddar, A., Rezagholizadeh, M., Langlais, P., & Parthasarathi, P. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.12
MuLER: Detailed and Scalable Reference-based Evaluation
Karidi, T., Choshen, L., Patel, G., & Abend, O. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.29
Learning to Rank Generation with Pairwise Partial Rewards
Lee, Y., Lee, J., & Hwang, S.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.371
Complementary Roles of Inference and Language Models in QA
Cheng, L., Hosseini, M. J., & Steedman, M. J. (2023) · doi:10.18653/v1/2023.pandl-1.8
DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization
Hu, C., Liu, X., & Feng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.924
Emergence of Abstract State Representations in Embodied Sequence Modeling
Yun, T., Zeng, Z., Handa, K., Thapliyal, A., Pang, B., Pavlick, E., & Sun, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.749 · Cited by 6
Boosting Summarization with Normalizing Flows and Aggressive Training
Yu, Y. F., & Shen, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.165 · Cited by 1
Post-hoc Utterance Refining Method by Entity Mining for Faithful Knowledge Grounded Conversations
Jang, Y., Son, S., Lee, J., Son, J., Hur, Y., Lim, J., Moon, H., Yang, K., & Lim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.295
Harnessing Black-Box Control to Boost Commonsense in LM’s Generation
Tian, Y., Zhang, F., & Peng, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.329 · Cited by 1
EELBERT: Tiny Models through Dynamic Embeddings
Cohn, G., Agarwal, R., Gupta, D., & Patwardhan, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.43 · Cited by 2
Remember what you did so you know what to do next
Ciosici, M. R., Hedges, A., Kankanampati, Y., Martin, J. D., Freedman, M., & Weischedel, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.104
kNN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers
Bhardwaj, R., Li, Y., Majumder, N., Cheng, B. H., & Poria, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.903 · Cited by 1
Huang, J., Yao, W., Song, K., Zhang, H., Chen, M., & Yu, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.900 · Cited by 1
Simple and Effective Input Reformulations for Translation
Yu, B., Lillemark, H. J., & Keutzer, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.638
Abstractive Open Information Extraction
Pei, K., Jindal, I., & Chang, K. C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.376
EntSUMv2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization
Mehra, D., Xie, L., Hofmann-Coyle, E., Kulkarni, M., & Preoțiuc-Pietro, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.337
When Do Decompositions Help for Machine Reading?
Wei, K., Lawrie, D., Durme, B. V., Chen, Y., & Weller, O. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.219 · Cited by 2
Too much of product information : Don’t worry, let’s look for evidence!
Jain, A., Rana, J. B., & Aggarwal, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.68
Yokoyama, T., Natter, J., & Godet, J. (2023) · medRxiv · doi:10.1101/2023.12.09.23299754 · Cited by 1
Rong, H., Yu, X., Ma, T., Sheng, V. S., Zhou, Y., & Al‐Rodhaan, M. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3636511
Self-citationImproving Chinese-Centric Low-Resource Translation Using English-Centric Pivoted Parallel Data
wang, X., Mu, L., & Xu, H. (2023) · International Conference on Asian Language Processing · doi:10.1109/ialp61005.2023.10337297
An Improved Framework for Scaling Party Positions from Texts with TransformerPreprint
Nguyen, H. H. V. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/8sha3
Interactive Task Learning for Social Robots: A Pilot Study
Tyshka, A., & Louie, W.-Y. G. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341713 · Cited by 1
Using Language Models to Augment Data in Stock Prediction
Hou, Y. (2023) · Advances in Economics and Management Research · doi:10.56028/aemr.8.1.298.2023
Tracing and Manipulating Intermediate Results in Neural Math Problem Solvers
Matsumoto, Y., Heinzerling, B., Yoshikawa, M., & Inui, K. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.1151
Mitsuda, K., Higashinaka, R., Li, T., Sugiyama, H., Mizukami, M., Nakamura, R., Adachi, N., Kawabata, H., Yoshida, S., & Kinebuchi, T. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.1172
Longjaroen, W., Chay-intr, T., Funakoshi, K., et al. (2023) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp60301.2023.10355030
Triple Extraction with Generative Technique for Constructing Weighted Knowledge Graph
Parniani, M. S., & Reformat, M. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00023
Fine-tuning Transformer-based MT Using Syntactic Guides
Makwisai, J., Boonkwan, P., Usanavasin, S., Kaothanthong, N., & Okumura, M. (2023) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp60301.2023.10354857
Capturing natural position relationships: A neural differential equation approach
Ji, C., Wang, L., Wang, L., et al. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.12.006
Application of Noise Filter Mechanism for T5-Based Text-to-SQL Generation
Rushdy, M. R. A., & Thayasivam, U. (2023) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon60487.2023.10355492 · Cited by 1
An Intelligent English Teaching System for Humanoid Applications
Yang, Q., Guan, Y., Chen, W., & Chen, W. (2023) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio58561.2023.10355010 · Cited by 2
Navigating GPT-4 and BERT: A Dual Perspective on Financial and Political Sentiment Analysis
Ghosh, A., Biswas, D., Sarkar, R., Pillai, A., & Chopra, J. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.57687
Mucha: Multi-channel based Code Change Representation Learning for Commit Message Generation
Wang, C., Wu, Y., & Zhang, X. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00064
DeCAB: Debiased Semi-supervised Learning for Imbalanced Open-Set Data
Huang, X., Li, M., Yang, L., & Wang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8546-3_9
Enhanced protein function prediction by fusion embedding based on protein language models
Wang, Y. (2023) · Highlights in Science Engineering and Technology · doi:10.54097/hset.v66i.11697
Summarization of Investment Reports Using Pre-trained Model
Sakaji, H., Kobayashi, R., Izumi, K., Mitsugi, H., & Kuramoto, W. (2023) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai59060.2023.00111
Ramyasree, K., & Kumar, C. S. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0141281
Online Behaviour Indicator Extraction for Enhanced Cancer Patient Management using Real-World Data
Pantalona, G., Barachanou, A., Loukas, N., Apostolidis, L., Tsalakanidou, F., & Papadopoulos, S. (2023) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1145/3617233.3617264
Multi-Task Learning of Japanese How-to Tip Machine Reading Comprehension by a Generative Model
Wang, X., Li, T., TAMURA, T., NISHIDA, S., & Utsuro, T. (2023) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2023edp7113
T4S: Two-Stage Screenplay Synopsis Summary Generation with Turning Points
Wang, D., Sun, W., Luo, C., Liu, D., Mao, R., & Xu, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-51671-9_6
Adaptive Sentence Selection for Extractive Summarization: Leveraging Dynamic Extraction StrategiesPreprint
Wei, H., Zhang, L., Yan, Y., & Pan, H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4682768
Sujatha, R., & Nimala, K. (2023) · International Conference on Advances in Electrical, Electronics and Computational Intelligence (ICAEECI) · doi:10.1109/icaeeci58247.2023.10370954
Prompt-Based and Two-Stage Training for Few-Shot Text Classification
Yan, Z., Tang, Y., & Liu, X. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9637-7_2
Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models
Kiyono, S., Takase, S., Li, S., & Sato, T. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_062
MQDD: Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain
Pašek, J., Sido, J., Konopík, M., & Pražák, O. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_089 · Cited by 3
Publish or Hold? Automatic Comment Moderation in Luxembourgish News Articles
Ranasinghe, T., Plum, A., Purschke, C., & Zampieri, M. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_104
Genome-scale annotation of protein binding sites via language model and geometric deep learningPreprint
Yuan, Q., Tian, C., & Yang, Y. (2024) · eLife · doi:10.7554/elife.93695.1
Cação, F. N. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.3.2023.tde-10012024-092904
Enhancing Person Re-identification Across Modalities with Local and Global Feature Correlation
Li, L., Wu, N., Wang, Z., & Liu, X. (2023) · International Conference on Computational Intelligence and Applications (ICCIA) · doi:10.1109/iccia59741.2023.00039
BrainPST: Pre-training Stacked Transformers for Dynamic Brain Functional Network Analysis
Hu, J., Huang, Y., Zhuo, Y., & Dong, S. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385467
TS-DDD: A Two-Stage Training Strategy for Dialogue-based Disease Diagnosis
Zhang, T., Cao, H., Feng, S., Cui, Y., & Wang, D. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385269
Sunpheno: a deep neural network for phenological classification of sunflower imagesPreprint
Luoni, S. A. B., Ricci, R., Corzo, M. A., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.17.574407
(2024) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets48442
Introduction to machine learning and its applications in stem cell research
Raghav, N., K., A. V. G., Deshpande, N., & Rangarajan, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13222-3.00025-3
Evaluating CLIP’s Understanding on Relationships in a Blocks World
Zhang, K., & Lewis, M. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386915
Tekumalla, R., Hernandez, L. A. R., & Banda, J. M. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386536
An Interactive Framework of Balancing Evaluation Cost and Prediction Accuracy for Knowledge Tracing
Sun, W., Gao, B., Chen, J., Huang, S., & Luo, W. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386686
Language-Model-based methods for Vietnamese Single-Document Extractive Summarization
Cuong, V. L., Thịnh, V. X., Thien, V. D., Hung, V. S., & Viết, N. X. (2023) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc58733.2023.10392644
Yun, J., Kim, S., Jung, M., et al. (2023) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc58733.2023.10393234
Phylogeny-Inspired Soft Prompts For Data-to-Text Generation in Low-Resource Languages
Martinez, W. S., Parmentier, Y., & Gardent, C. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.13 · Cited by 1
Improving Code Summarization Performance with Model Fusion
Guo, Y., Su, H., Gao, H., & Sun, Q. (2023) · International Conference on Intelligent Design (ICID) · doi:10.1109/icid60307.2023.10396770
Recognizing Textual Inference in Mongolian Bar Exam Questions
Khaltarkhuu, G., Batjargal, B., & Maeda, A. (2024) · Applied Sciences · doi:10.3390/app14031073
Semi-Supervised Learning for Prompt Classification in ChatGPT
Ilie, A.-G., & Rebedea, T. E. (2023) · RoCHI - International Conference on Human-Computer Interaction (RoCHI) · doi:10.37789/rochi.2023.1.1.9
A Dynamic Global Semantic Fusion GNN Model For Commonsense Question Answering
Li, J., Wang, G., Tian, C., & Liu, S. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394522
Taking Natural Language Generation and Information Extraction to Domain Specific Tasks
Varma, S., Shivam, S., Natarajan, S., et al. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47715-7_48
VietLegalQA: Unsupervised Legal Question Answering for Vietnamese Using Cloze Translation Approach
Ngoc, K. N., Trong, P. N., Minh, C. L. N., Khang, N. H. G., Quoc, T. N., & Hoang, V. T. (2023) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn59264.2023.10402152
Diversity question generation based on contrastive search algorithmPreprint
Zhou, M., Zhou, J., Gan, J., Wang, J., & Wang, M. (2024) · Research Square · doi:10.21203/rs.3.rs-3887342/v1
Deep Learning in Paraphrase Generation: A Systematic Literature Review
Natsir, A. H., Hidayah, I., & Adji, T. B. (2023) · International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE) · doi:10.1109/icitisee58992.2023.10405123 · Cited by 1
Metadata
- Authors
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Volume
- 21
- Issue
- 140
- Pages
- 1-67
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.10683
- OpenAlex
- W2981852735
- Cited by
- 10,929
- References
- 125
- Citations by year
- 2026: 57 · 2025: 1758 · 2024: 2734 · 2023: 3457 · 2022: 1678 · 2021: 833 · 2020: 399 · 2019: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Journal of Machine Learning Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transfer learning, text-to-text transformers, pre-training objectives, pre-trained language models, language understanding tasks, summarization, question answering, text classification, fine-tuning, unlabeled data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
