The DOI matches its Crossref record; 57 of the 63 references it cites were found in a registry or an index, and 6 are not checked yet; none of the 16 sources its claims cite is checked for support yet.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Open accessAbstract
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.
Checked 2026-11-10 · 6 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper studies continued pretraining of RoBERTa on domain-specific and task-specific unlabeled text across four domains and eight classification tasks. It finds that domain-adaptive pretraining (DAPT) and task-adaptive pretraining (TAPT) both improve performance, with DAPT followed by TAPT giving the best results.
Key findings
- DAPT improved over RoBERTa on all eight tasks, including CHEMPROT (81.9 → 84.2 microF1), ACL-ARC (63.0 → 75.4 macroF1), SCIERC (77.3 → 80.8), HYPERPARTISAN (86.6 → 88.2), and HELPFULNESS (65.1 → 66.5); AGNEWS was unchanged at 93.9.
- TAPT consistently improved over RoBERTa for all tasks, and DAPT + TAPT achieved the best performance on all tasks, e.g. RCT 87.8, ACL-ARC 75.6, SCIERC 81.3, HYPERPARTISAN 90.0, AGNEWS 94.6, HELPFULNESS 68.7, IMDB 95.6.
- Curated-TAPT further improved over TAPT on RCT-500 (79.8 → 83.4), HYPERPARTISAN (90.4 → 89.9), and IMDB (95.5 → 95.7); DAPT + Curated-TAPT gave the largest boosts, e.g. RCT-500 83.8 and HYPERPARTISAN 92.1.
- Automated kNN-TAPT outperformed TAPT for all cases, with 500 NN-TAPT reaching CHEMPROT 83.3, RCT-500 81.7, and ACL-ARC 75.5; TAPT was nearly 60 times faster to train than DAPT, while DAPT storage was 5.8M times that of TAPT on RCT-500.
Methods
- The authors continued pretraining RoBERTa on four domains: biomedical papers (7.55B tokens), CS papers (8.10B tokens), news from RealNews (6.66B tokens), and Amazon reviews (2.11B tokens), then evaluated on eight classification tasks in high- and low-resource settings.
- DAPT trained for 12.5K steps per domain; TAPT trained on the unlabeled task training set for 100 epochs with 0.15 masking; DAPT + TAPT applied both sequentially. Classification used the final-layer [CLS] representation with a task-specific feedforward layer, reporting test macroF1 or microF1 averaged over five random seeds.
- For automated data selection, they embedded task and domain sentences with VAMPIRE, selected k nearest neighbors from the domain corpus, and continued pretraining RoBERTa on the task data plus selected candidates.
Limitations
- The paper states that cross-DAPT masked LM loss can be a noisy predictor of domain similarity, that domain boundaries are fuzzy, and that TAPT optimizes for single-task performance to the detriment of cross-task transfer. It also notes high variance on HYPERPARTISAN due to its very small dataset, and that degenerate seeds were discarded and resampled.
- The paper states that BIOMED and CS used an internal version of S2ORC containing papers that cannot be released due to copyright restrictions.
- Evident from the text: the study is limited to RoBERTa, four English-language domains, and eight classification tasks; the authors call for future work on better data selection, efficient adaptation to distant domains, and reusable adapted language models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
37 Own result · 19 Own interpretation · 9 No basis given · 8 Not checked yet · 4 Own method · 1 Sources not found · 1 Check pending
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that TAPT provides a large performance boost for ROBERTA, with or without domain-adaptive pretraining.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that the cross-domain masked-LM loss mostly follows our intuition and insights from the paper, i.e. ROBERTA's pretraining corpus and NEWS are closer, and BIOMED to CS (relative to other domains).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, ROBERTA that is adapted to REVIEWS results in the highest loss for a NEWS sample.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there are two tasks (SCIERC and ACL-ARC) in which ¬ DAPT marginally improves performance over ROBERTA.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TAPT consistently improves the ROBERTA baseline for all tasks across domains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automatic methods such as k NN-TAPT are much cheaper than DAPT.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For each task, DAPT significantly outperforms adapting to an irrelevant domain, suggesting the importance of pretraining on domain-relevant data.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results in Table 8 show that k NN-TAPT outperforms TAPT for all cases.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that the cross-domain masked-LM loss mostly follows our intuition and insights from the paper, i.e. ROBERTA's pretraining corpus and NEWS are closer, and BIOMED to CS (relative to other domains).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying Curated-TAPT after adapting to the domain results in the largest boost in performance on all tasks”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 18
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is the case for all domains.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we increase k , k NN-TAPT performance steadily increases, and approaches that of DAPT.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 20 we display additional examples that highlight the overlap between IMDB reviews and REALNEWS articles, relevant for analysis in § 3.1.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RAND-TAPT is generally worse than k NN-TAPT, but within a standard deviation arising from 5 seeds for RCT and ACLARC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the gradations in Figure 2 suggest that the boundaries between domains are in some sense fuzzy; for example, 40% of unigrams are shared between REVIEWS and NEWS.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One of the properties that distinguishes REVIEWS from all other domains is that its documents are significantly shorter.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our baseline establishes the first benchmark for the HELPFULNESS dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“AMAZON reviews contain more documents, but each is shorter.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that masked LM loss decreases in all domains except NEWS after DAPT, where we observe a marginal increase.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, ROBERTA is not drastically behind the state of the art (details in Appendix § A.2), and serves as a good baseline since it provides a single LM to adapt to different domains.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For BIOMED, CS, and REVIEWS, we see consistent improvements over ROBERTA, demonstrating the benefit of DAPT when the target domain is more distant from ROBERTA's source domain.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TAPT is nearly 60 times faster to train than DAPT on a single v3-8 TPU and storage requirements for DAPT on this task are 5.8M times that of TAPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that ROBERTA's pretraining domain has strong vocabulary overlap with NEWS and REVIEWS, while CS and BIOMED are far more dissimilar to the other domains.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining indomain ( domain-adaptive pretraining ) leads to performance gains, under both high- and low-resource settings.”
- Also stated in: § 3.2 Experiments · p. 3 · plain
“We observe that DAPT improves over ROBERTA in all domains.”
- Also stated in: § 3.2 Experiments · p. 3 · plain
“The pattern is consistent across high- and low- resource settings.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 3.2 Experiments · p. 3 · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We report averages across five random seeds, with standard deviations as subscripts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which suggests a spectra of domains defined around tasks at various levels of granularity (e.g., Amazon reviews for a specific product, all Amazon reviews, all reviews on the web, the web).”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“ROBERTA was trained on data from BOOKCORPUS (Zhu et al., 2015), 6 WIKIPEDIA, 7 a portion of the CCNEWS dataset (Nagel, 2016), 8 OPENWEBTEXT corpus of Web content extracted from URLs shared on Reddit (Gokaslan and Cohen, 2019), 9 and a subset of CommonCrawl that it is said to resemble the 'story-like' style of WINOGRAD schemas (STORIES; Trinh and Le, 2018).”
What the sources say
- Not checked yetcites Zhu et al., 2015 source
Checked before — its earlier verdict is held back until a new check runs.
- Not matchedcites Nagel, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Gokaslan and Cohen, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not checked yet
In the paper
“Adhering to the standards suggested by Dodge et al. (2019) for replication, we report our development set results in Tables 15, 17, and 18.”
What the sources say
- Not checked yetcites Dodge et al. (2019) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe a high variance in performance between random seeds when fine-tuning ROBERTA to HYPERPARTISAN, because the dataset is extremely small.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Prior work has shown the benefit of continued pretraining in domain (Alsentzer et al., 2019; Chakrabarty et al., 2019; Lee et al., 2019).”
What the sources say
- Not checked — no full text of the sourcecites Alsentzer et al., 2019 source
- Not checked yetcites Chakrabarty et al., 2019 source
Checked before — its earlier verdict is held back until a new check runs.
- Not matchedcites Lee et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“This may suggest that in some cases, continued pretraining on any additional data is useful, as noted in Baevski et al. (2019).”
What the sources say
- Not checked yetcites Baevski et al. (2019) source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, this could also explain why adapting only to a broad domain is not sufficient, and why TAPT after DAPT is effective.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“State-of-the-art results we can compare to: CHEMPROT (84.6), RCT (92.9), ACL-ARC (71.0), SCIERC (81.8), HYPERPARTISAN (94.8), AGNEWS (95.5), IMDB (96.2); references in § A.2.”
What the sources say
- Not matchedcites § A.2
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“ROBERTA attains better performance on an assortment of tasks than its predecessors, making it our baseline of choice.”
What the sources say
- Not checked yetcites Liu et al., 2019 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A check of a cited source is scheduled, or we don’t hold its full text yet; the claim is checked once it runs.
In the paper
“Continued pretraining of a LM on the unlabeled data of a given task (TAPT) has been show to be beneficial for endtask performance (e.g. in Howard and Ruder, 2018; Phang et al., 2018; Sun et al., 2019).”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Mean test set microF 1 (for CHEMPROT and RCT) and macroF 1 (for ACL-ARC), across five random seeds, with standard deviations as subscripts, comparing RAND-TAPT (with 50 candidates) and k NNTAPT selection.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While our results demonstrate how these approaches can improve ROBERTA, a powerful LM, the approaches we studied are general enough to be applied to any pretrained LM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, our work explores multiple domains, and is arguably more cost effective, since we continue pretraining an already powerful LM.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We focused on matching all the domain dataset sizes (see Table 1) such that each domain is exposed to the same amount of data as for 12.5K steps it is trained for.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Reported results are development macroF 1 , except for CHEMPROT and RCT, for which we report microF 1 , following Beltagy et al. (2019).”
What the sources say
- Not checked yetcites Beltagy et al. (2019) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To produce final results on this task, we discard and resample degenerate seeds.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“ULMFIT is pretrained on English Wikipedia; ULMFIT † on English tweets; ELMO on the 1BWORDBENCHMARK (newswire; Chelba et al., 2014); GPT on BOOKCORPUS; BERT on English Wikipedia and BOOKCORPUS.”
What the sources say
- Not checked yetcites Chelba et al., 2014 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Task-adaptive pretraining has been shown effective (Howard and Ruder, 2018), but is not typically used with the most recent models.”
- Also stated in: § 4 Task-Adaptive Pretraining · p. 5 · plain
“prior work has shown its effectiveness (e.g. Howard and Ruder, 2018).”
What the sources say
- Not checked yetcites Howard and Ruder, 2018 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § 4 Task-Adaptive Pretraining · p. 5 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Compared to domainadaptive pretraining (DAPT; § 3), the task-adaptive approach strikes a different trade-off: it uses a far smaller pretraining corpus, but one that is much more task-relevant (under the assumption that the training set represents aspects of the task well).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The training set represents aspects of the task well.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our analysis in § 3.1 illustrates that REVIEWS and NEWS also have some similarities.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
ROBERTA's pretraining sources may not be diverse enough to generalize to most of the variation in the English language.
“Although ROBERTA's pretraining corpus is derived from multiple sources, it has not yet been established if these sources are diverse enough to generalize to most of the variation in the English language.”
The authors state this as an assumption.
The training set represents aspects of the task well.
“under the assumption that the training set represents aspects of the task well”
The paper relies on this without support.
Task data is non-randomly sampled from a wider distribution within a larger target domain that is not necessarily one of the original LM pretraining domains.
“Task data is comprised of an observable task distribution, usually non-randomly sampled from a wider distribution (light grey ellipsis) within an even larger target domain, which is not necessarily one of the domains included in the original LM pretraining domain - though overlap is possible.”
The paper relies on this without support.
The larger unlabeled corpus collected for a task is expected to have a similar distribution to the task's training data.
“The larger unlabeled corpus is thus expected to have a similar distribution to the task's training data.”
The paper relies on this without support.
Such a larger unlabeled corpus is usually available.
“Moreover, it is usually available.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Domain adaptation studies [23] show that task performance varies across domains, with general-purpose LLMs exhibiting stronger capabilities in well-represented areas such as Computer Vision than in specialized fields such as Quantum Computing, due to differences in pretraining corpus composition.”
From Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Continued pre-training on domainspecific data has been shown to be effective for improving performance in a specific domain (Gururangan et al., 2020).”
From The Llama 3 Herd of Models (2024) · cites this paper as Gururangan et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Further pre-training has been shown to increase scores when adapting a model to a specific domain or task (Gururangan et al., 2020;Lee et al., 2020).”
From QueryNER: Segmentation of E-commerce Queries (2024) · cites this paper as Gururangan et al., 2020
In the citing paper
“Task adaptive pre-training (TAPT) and domain adaptive pre-training (DAPT) have been proposed by Gururangan et al. [37]. TAPT continues to train it on the target task's training dataset, after pre-training RoBERTa on large-scale corpora. DAPT is still pre-training RoBERTa on datasets with minor domain differences from datasets in target tasks.”
From Self-supervised learning: A succinct review (2023) · cites this paper as [37]
In the citing paper
“The method relies on a comparison of gridbox-scale trends in observational datasets for temperature (HadCRUT4 version 4.6 54 ) and precipitation (GPCC v2018, obtainable from https://psl.noaa.gov/data/gridded/data.gpcc.html),with those produced in climate model runs from CMIP6 55 .”
From Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies (2021) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ F k -Nearest Neighbors Data Selection · p. 18
“Table 19: Mean development set macroF 1 (for HYP. and IMDB) and microF 1 (for RCT), across five random seeds, with standard deviations as subscripts, comparing RAND-TAPT (with 50 candidates) and k NN-TAPT selection.”
§ F k -Nearest Neighbors Data Selection · p. 18
“| | CHEMPROT | RCT-500 | ACL-ARC |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 Computational Requirements · p. 8
“storage requirements for DAPT on this task are 5.8M times that of TAPT.”
§ 5.3 Computational Requirements · p. 8
“| TAPT | 0.2K | 500 | 80KB | 79.8 1 . 4 | | 50 NN-TAPT | 1.1K | 24K | 3MB | 80.8 0 . 6 | | 150 NN-TAPT | 3.2K | 66K | 8MB | 81.2 0 . 8 | | 500 NN-TAPT | 9.0K | 185K | 24MB | 81.7 0 . 4 | | Curated-TAPT | 8.8K | 180K | 27MB | 83.4 0 . 3 | | DAPT | 12.5K | 25M | 47GB | 82.5 0 . 5 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 Human Curated-TAPT · p. 7
“Curated-TAPT further improves our prior results from § 4 across all three datasets.”
§ 5 Augmenting Training Data for Task-Adaptive Pretraining · p. 7
“| TAPT | 79.8 1 . 4 | 90.4 5 . 2 | 95.5 0 . 1 | | DAPT + TAPT | 83.0 0 . 3 | 90.0 6 . 6 | 95.6 0 . 1 | | Curated-TAPT | 83.4 0 . 3 | 89.9 9 . 5 | 95.7 0 . 1 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 Human Curated-TAPT · p. 7
“We simulate a low-resource setting RCT500, by downsampling the training data of the RCT dataset to 500 examples (out of 180K available), and treat the rest of the training data as unlabeled.”
§ 3.2 Experiments · p. 4
“| BIOMED | CHEMPROT † RCT | relation classification abstract sent. roles | 4169 18040 | - - | 2427 30212 | 3469 30135 | 13 5 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.2 Automated Data Selection for TAPT · p. 8
“As we increase k , k NN-TAPT performance steadily increases, and approaches that of DAPT.”
§ 5.1 Human Curated-TAPT · p. 7
“| 50 NN-TAPT | 83.3 0 . 7 | 80.8 0 . 6 | 70.7 2 . 8 | | 150 NN-TAPT | 83.2 0 . 6 | 81.2 0 . 8 | 73.3 2 . 7 | | 500 NN-TAPT | 83.3 0 . 7 | 81.7 0 . 4 | 75.5 1 . 9 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
Research works printed but not linked: 5
- Franck Dernoncourt and Ji Young Lee. 2017. Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In IJCNLP . https://www.aclweb.org/anthology/I17-2052/
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In ICLR . To appear.
- Martin Krallinger, Obdulia Rabal, Saber Ahmad Akhondi, Mart´ ın P´ erez P´ erez, J´ es ´ us L´ opez Santamar´ ıa, Gael P´ erez Rodr´ ıguez, Georgios Tsatsaronis, Ander Intxaurrondo, Jos´ e Antonio Baso L´ opez, Umesh Nandal, E. M. van Buel, A. Poorna Chandrasekhar, Marleen Rodenburg, Astrid Lægreid, Marius A. Doornenbal, Julen Oyarz´ abal, An´ alia Lourenc ¸o, and Alfonso Valencia. 2017. Overview of the biocreative vi chemical-protein interaction track. In Proceedings of the BioCreative VI Workshop .
- Steffen Remus and Chris Biemann. 2016. DomainSpecific Corpus Expansion with Focused Webcrawling. In LREC .
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS .
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,694
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
BERTweet: A pre-trained language model for English Tweets
Nguyen, D. Q., Vu, T., & Nguyen, A. T. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.2 · Cited by 1,050
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 1,324
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
BERTimbau: Pretrained BERT Models for Brazilian Portuguese
Souza, F., Nogueira, R. F., & Lotufo, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61377-8_28 · Cited by 751
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Self-citationSUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681
AdapterHub: A Framework for Adapting Transformers
Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.7 · Cited by 613
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 430
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382
Self-citationHateBERT: Retraining BERT for Abusive Language Detection in English
Caselli, T., Basile, V., Mitrović, J., & Granitzer, M. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.3 · Cited by 373
MatSciBERT: A materials domain language model for text mining and information extraction
Gupta, T., Zaki, M. F. M., Krishnan, N. M. A., & Mausam, M. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00784-w · Cited by 388
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.201 · Cited by 310
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306
Self-citationMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Lawformer: A pre-trained language model for Chinese legal long documents
Xiao, C., Hu, X., Liu, Z., Tu, C., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.003 · Cited by 324
Self-supervised learning: A succinct review
Rani, V., Nabi, S. T., Kumar, M., Mittal, A., & Kumar, K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09884-2 · Cited by 295
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Should ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 292
MasakhaNER: Named Entity Recognition for African Languages
Adelani, D. I., Abbott, J., Neubig, G., D’souza, D., Kreutzer, J., Lignos, C., Palen-Michel, C., Buzaaba, H., Rijhwani, S., Ruder, S., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00416 · Cited by 294
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 328
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Sharma, A., Miner, A., Atkins, D., & Althoff, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.425 · Cited by 268
Neural Unsupervised Domain Adaptation in NLP—A Survey
Ramponi, A., & Plank, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.603 · Cited by 311
Domain Adaptation: Challenges, Methods, Datasets, and Applications
Singhal, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2023) · IEEE Access · doi:10.1109/access.2023.3237025 · Cited by 276
Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies
Callaghan, M., Schleussner, C.-F., Nath, S., Lejeune, Q., Knutson, T. R., Reichstein, M., Hansen, G., Theokritoff, E., Andrijevic, M., Brecha, R. J., et al. (2021) · Nature Climate Change · doi:10.1038/s41558-021-01168-6 · Cited by 251
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 357
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare
Ji, S., Zhang, T., Ansari, L., Fu, J., Tiwari, P., & Cambria, E. (2022) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/2d7umxifj8wz · Cited by 233
CLIMATEBERT: A Pretrained Language Model for Climate-Related TextPreprint
Webersinke, N., Kraus, M., Bingler, J., & Leippold, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4229146 · Cited by 313
Can Foundation Models Wrangle Your Data?
Narayan, A., Chami, I., Orr, L., & Ré, C. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3574245.3574258 · Cited by 329
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590
“Similar to the don’t stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don’t stop instruction-tuning to build customized LMM. ( iii ) Low serving cost.”
BackgroundSelf-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227
Singh, R., Sledzieski, S., Bryson, B. D., Cowen, L., & Berger, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2220778120 · Cited by 216
An Analysis of Simple Data Augmentation for Named Entity Recognition
Dai, X., & Adel, H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.343 · Cited by 230
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 238
COVIDLies: Detecting COVID-19 Misinformation on Social Media
Hossain, T., Logan, R. L., Ugarte, D. A., Matsubara, Y., Young, S. D., & Singh, S. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.11 · Cited by 214
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202
Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited
Yuan, Z., Yuan, F., Song, Y., Li, Y., Fu, J., Yang, F., Pan, Y., & Ni, Y. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591932 · Cited by 200
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 320
Self-citationChristou, P. (2023) · The Qualitative Report · doi:10.46743/2160-3715/2023.6406 · Cited by 226
Continual Lifelong Learning in Natural Language Processing: A Survey
Biesialska, M., Biesialska, K., & Costa‐jussà, M. R. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.574 · Cited by 244
Knowledge Graph Prompting for Multi-Document Question Answering
Wang, Y., Lipka, N., Rossi, R. A., Siu, A. F., Zhang, R., & Derr, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29889 · Cited by 233
Wang, K., Reimers, N., & Gurevych, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.59 · Cited by 228
End-to-End Transformer-Based Models in Textual-Based NLP
Rahali, A., & Akhloufi, M. A. (2023) · AI · doi:10.3390/ai4010004 · Cited by 209
Unsupervised Domain Clusters in Pretrained Language Models
Aharoni, R., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.692 · Cited by 181
Lewis, P., Ott, M., Du, J., & Stoyanov, V. (2020) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2020.clinicalnlp-1.17 · Cited by 196
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.544 · Cited by 185
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
He, R., Liu, L., Ye, H., Tan, Q., Ding, B., Cheng, L., Low, J.-W., Bing, L., & Si, L. (2021) · doi:10.18653/v1/2021.acl-long.172 · Cited by 222
SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge
Ke, P., Ji, H., Liu, S., Zhu, X., & Huang, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.567 · Cited by 190
Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition
Chen, L.-W., & Rudnicky, A. I. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095036 · Cited by 185
Koto, F., Lau, J., & Baldwin, T. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.833 · Cited by 192
Investigating Pretrained Language Models for Graph-to-Text Generation
Ribeiro, L. F. R., Schmitt, M., Schütze, H., & Gurevych, I. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.20 · Cited by 160
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Word Alignment by Fine-tuning Embeddings on Parallel Corpora
Dou, Z.-Y., & Neubig, G. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.181 · Cited by 172
CrossNER: Evaluating Cross-Domain Named Entity Recognition
Liu, Z., Xu, Y., Yu, T., Dai, W., Ji, Z., Cahyawijaya, S., Madotto, A., & Fung, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17587 · Cited by 161
BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
Yuan, H., Yuan, Z., Gan, R., Zhang, J., Xie, Y., & Yu, S. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.9 · Cited by 178
A comprehensive survey on machine learning approaches for fake news detection
Alghamdi, J., Luo, S., & Lin, Y. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17470-8 · Cited by 157
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 159
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Mari Ostendorf, E., Yih, W.-T., Smith, N. A., Zettlemoyer, L., & Yu, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.71 · Cited by 190
Self-citationMTP: Advancing Remote Sensing Foundation Model via Multitask Pretraining
Wang, D., Zhang, J., Xu, M., Liu, L., Wang, D., Gao, E., Han, C., Guo, H., Du, B., Tao, D., et al. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3408154 · Cited by 170
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 146
A Comparative Study of Machine Learning and Deep Learning Techniques for Fake News Detection
Alghamdi, J., Lin, Y., & Luo, S. (2022) · Information · doi:10.3390/info13120576 · Cited by 133
Learning to Pre-train Graph Neural Networks
Lu, Y., Jiang, X., Fang, Y., & Shi, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16552 · Cited by 128
Improving and Simplifying Pattern Exploiting Training
Tam, D., Menon, R. R., Bansal, M., Srivastava, S., & Raffel, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.407 · Cited by 145
Towards COVID-19 fake news detection using transformer-based models
Alghamdi, J., Lin, Y., & Luo, S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110642 · Cited by 130
Systematic review of data-centric approaches in artificial intelligence and machine learning
Singh, P. (2023) · Data Science and Management · doi:10.1016/j.dsm.2023.06.001 · Cited by 136
Merging Models with Fisher-Weighted Averaging
Matena, M. S., & Raffel, C. (2022) · neural information processing systems · doi:10.52202/068431-1287 · Cited by 206
Table-GPT: Table Fine-tuned GPT for Diverse Table Tasks
Li, P., He, Y., Yashar, D., Cui, W., Song, G., Zhang, H., Fainman, D. R., Zhang, D., & Chaudhuri, S. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654979 · Cited by 194
CLEVE: Contrastive Pre-training for Event Extraction
Wang, Z., Wang, X., Han, X., Lin, Y., Hou, L., Liu, Z., Li, P., Li, J., & Zhou, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.491 · Cited by 126
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 145
Addressing bias in generative AI: Challenges and research opportunities in information management
Wei, X., Kumar, N., & Zhang, H. (2025) · Information & Management · doi:10.1016/j.im.2025.104103 · Cited by 151
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented DialoguePreprint
Mehri, S., Eric, M., & Hakkani-Tür, D. Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13570 · Cited by 141
Generative Data Augmentation for Commonsense Reasoning
Yang, Y., Malaviya, C., Fernandez, J., Swayamdipta, S., Bras, R. J. L., Wang, J., Bhagavatula, C., Choi, Y., & Downey, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.90 · Cited by 156
Self-citationTowards Geospatial Foundation Models via Continual Pretraining
Mendieta, M., Han, B., Shi, X., Zhu, Y., & Chen, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01541 · Cited by 157
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
LogFiT: Log Anomaly Detection Using Fine-Tuned Language Models
Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3358730 · Cited by 111
LAMBERT: Layout-Aware Language Modeling for Information Extraction
Garncarek, Ł., Powalski, R., Stanisławek, T., Topolski, B., Halama, P., Turski, M., & Graliński, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86549-8_34 · Cited by 120
CausaLM: Causal Model Explanation Through Counterfactual Language Models
Feder, A., Oved, N., Shalit, U., & Reichart, R. (2021) · Computational Linguistics · doi:10.1162/coli_a_00404 · Cited by 119
Self-Supervised Pretraining Improves Self-Supervised Pretraining
Reed, C., Yue, X., Nrusimha, A., Ebrahimi, S., Vijaykumar, V., Mao, R., Li, B., Zhang, S., Guillory, D., Metzger, S. L., et al. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00112 · Cited by 116
Rethinking embedding coupling in pre-trained language modelsPreprint
Chung, H. W., Févry, T., Tsai, H., Johnson, M., & Ruder, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12821 · Cited by 125
Patwa, P., Bhardwaj, M., Guptha, V., Kumari, G., Sharma, S., Pykl, S., Das, A., Ekbal, A., Akhtar, M. S., & Chakraborty, T. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-73696-5_5 · Cited by 103
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.859 · Cited by 100
Improved Text Classification via Contrastive Adversarial Training
Pan, L., Hang, C.-W., Sil, A., & Potdar, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21362 · Cited by 105
Yu, Y., Zuo, S., Jiang, H., Ren, W., Zhao, T., & Zhang, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.84 · Cited by 118
PLM-ICD: Automatic ICD Coding with Pretrained Language Models
Huang, C.-W., Tsai, S.-C., & Chen, Y.-N. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.2 · Cited by 119
Mining legal arguments in court decisions
Habernal, I., Faber, D. R., Recchia, N., Bretthauer, S., Gurevych, I., Döhmann, I. S. G., & Burchard, C. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09361-y · Cited by 108
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 132
Metadata
- Authors
- Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 8342-8360
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3017961061
- Cited by
- 2,103
- References
- 62
- Citations by year
- 2026: 246 · 2025: 313 · 2024: 253 · 2023: 443 · 2022: 331 · 2021: 403 · 2020: 111
- FWCI
- 122.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- domain-adaptive pretraining, language model fine-tuning, text classification, resource-limited settings, data selection, biomedical text, computer science literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
