The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 10 of the 21 sources its claims cite were checked for support.
TLDR: Extreme Summarization of Scientific Documents
Open accessAbstract
We introduce TLDR generation, a new form of extreme summarization, for scientific papers.TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.To facilitate study on this task, we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers.SCITLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal.CATTS improves upon strong baselines under both automated metrics and human evaluations.Data and code are publicly available at https://github.com/allenai/scitldr. Dataset constructionOverview We introduce SCITLDR, a new multitarget dataset of 5,411 TLDRs over 3,229 scientific papers in the computer science domain.3 The training set contains 1,992 papers, each with a single gold TLDR.The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.This is unlike the majority of existing 3 See Appendix Table 9 for full venue breakdown.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces TLDR generation for scientific papers and releases SciTLDR, a multi-target dataset of 5,411 TLDRs over 3,229 computer science papers. It proposes CATTS, which uses paper titles as an auxiliary training signal, and reports improvements over strong extractive and abstractive baselines on automated and human evaluations.
Key findings
- SciTLDR contains 5,411 TLDRs over 3,229 papers: 1,992 training papers with one gold TLDR each, 619 dev papers with 1,452 TLDRs, and 618 test papers with 1,967 TLDRs. Its summaries average 21 words, with a compression ratio of 238.1 and 15.2% novel words.
- Author-written TLDRs average 18.9 words and peer-review-derived TLDRs average 22.9 words; lexical overlap is very low (1-, 2-, and 3-gram mean Jaccard indices of 15.0%, 2.5%, and 0.7%). TLDR-PR is more abstractive, with 20.2% novel words versus 9.6% for TLDR-Auth.
- CATTS improves over BART baselines: in abstract-only input, CATTS gains +0.5 Rouge-1 and CATTS_XSUM gains +1.8 Rouge-1; in AIC input, CATTS gains +2.0 Rouge-1 and CATTS_XSUM gains +0.9 Rouge-1. In AIC, CATTS reaches 44.9 Rouge-1, 22.6 Rouge-2, and 37.3 Rouge-L versus BART’s 42.9/20.8/35.1.
- Human evaluation found CATTS_XSUM more informative than BART_XSUM (MRR 0.54 vs 0.42), comparable to gold TLDR-Auth (0.53) but below TLDR-PR (0.60). Correctness did not differ: both models averaged 2.5, with 42 ties, 10 cases favoring BART_XSUM, and 12 favoring CATTS_XSUM.
Methods
- Dataset construction combined author-written TLDRs collected from OpenReview with expert-derived TLDR-PR summaries created by rewriting the first 128 words of peer review comments. Twenty-eight University of Washington undergraduate computer science students were recruited, 20 continued after initial training, and annotators were paid $20/hour; full texts were parsed with S2ORC.
- Baselines included extractive PACSUM, BERTSUMEXT, and MatchSum, plus abstractive BART-large and BART-large finetuned on XSUM. CATTS appends control codes for TLDR and title generation, trains with an additional 20K arXiv paper-title pairs, and upsamples TLDR instances; experiments use abstract-only and abstract/introduction/conclusion (AIC) input spaces.
- Evaluation uses max Rouge over multiple gold TLDRs per paper, plus human evaluation: nugget-based informativeness on 76 sampled gold papers and author-rated correctness from 29 unique authors covering 64 arXiv papers.
Limitations
- Stated in the paper: experiments are limited to abstract-only and AIC input spaces, though full texts are provided for future work; the idea of a TLDR may differ across academic disciplines and is left for future work; a small fraction (<5%) of papers lacked available PDFs and were included using abstract-only text.
- Evident from the text: the dataset is restricted to computer science papers, and the human correctness evaluation is small (29 authors, 64 papers) and found no difference between BART_XSUM and CATTS_XSUM.
- Evident from the text: informativeness analysis uses only 76 sampled gold papers, and automated evaluation relies on max Rouge over multiple gold TLDRs, which rewards matching any gold summary and may not fully capture factual correctness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
35 Own result · 15 No basis given · 9 Own method · 9 Own interpretation · 5 Partly supported · 4 Sources not found · 2 Supported · 2 No full text of the sources · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As evidenced by earlier work in summarization evaluation (Cohan and Goharian, 2016), variability in human-written summaries (Zechner, 1996; Harman and Over, 2004) can negatively impact the reliability of automated summarization metrics like Rouge (Lin, 2004).”
What the sources say
- Not matchedcites Cohan and Goharian, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Zechner, 1996 source
Section 4.3 reports mean interhuman precision and recall of 0.49 when each subject’s selected sentences were compared with the others, showing variability among human-produced extracts. The citation to Zechner is attached to that variability; the sentence cites other works for the evaluation and Rouge points. Page numbers are not available in the supplied copy. (LLM, conf=0.87)
- Not matchedcites Lin, 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“We demonstrate that control codes can be used to effectively incorporate scaffold tasks (e.g. title generation) for denoising autoencoders like BART (Lewis et al., 2020).”
What the sources say
- Supportedcites Lewis et al., 2020 source
Section 2 (Model; page not available) identifies BART as a denoising autoencoder, which is what the citation establishes in the citing sentence. The control-code result is presented as the citing authors’ own demonstration, not as a finding attributed to Lewis et al. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is unlike the majority of existing summarization datasets that assume only one gold summary for a given document.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also observe through qualitative examples in Figure 3 how TLDR-Auth and TLDR-PR can differ greatly, even when they contain the same information content.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, TLDRPR focus more on articulating problems the paper addresses ( P ).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The training set contains 1,992 papers, each with a single gold TLDR.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interestingly, TLDR-PR place less emphasis on R and V in favor of further methodological details in the paper D .”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers.”
- Also stated in: § 1 Introduction · p. 1 · plain
“we introduce SCITLDR, a new dataset of 5,411 TLDRs of computer science papers.”
- Also stated in: § 2 Dataset construction · p. 2 · plain
“We introduce SCITLDR, a new multitarget dataset of 5,411 TLDRs over 3,229 scientific papers in the computer science domain.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, they are the most frequently co-occurring nuggets, appearing in 63% of TLDRAuth and 71% of TLDR-PR.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We introduce CATTS (Controlled Abstraction for TLDRs with Title Scaffolding), a simple yet effective method for learning to generate TLDRs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Titles often contain key information about a paper.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 28
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The goal of existing work in summarization of scientific documents is to generate abstracts or provide complimentary summaries to abstracts. (Collins et al., 2017; Cohan et al., 2018; Chandrasekaran et al., 2019; Yasunaga et al., 2019).”
What the sources say
- Not matchedcites Collins et al., 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Cohan et al., 2018 source
The Introduction describes this paper’s model as generating summaries of scientific papers. The Data section says the papers’ abstracts serve as ground-truth summaries, supporting the claim’s abstract-generation alternative. (LLM, conf=0.89)
- Partly supportedcites Chandrasekaran et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes citation-based mini-summaries, and the Task section (p. 2) asks systems to generate a structured summary of a scientific paper. It lists abstracts alongside community and human-written summaries, but does not say that the task aims to generate abstracts or that its summaries are intended to complement abstracts. It also does not characterize the goals of existing work generally. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification (Swayamdipta et al., 2018b; Cohan et al., 2019).”
What the sources say
- Not matchedcites Swayamdipta et al., 2018b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Cohan et al., 2019 source
The Introduction and §2.1 describe auxiliary structural scaffold tasks in a multitask framework for citation-intent classification. The Results (§4.3) report improved classification performance when the scaffolds are added. This supports the text-classification part of the sentence for which Cohan et al. is cited. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“We use BART-large, which achieves state-of-the-art results in summarization on XSUM.”
What the sources say
- Supportedcites Lewis et al., 2020 source
Section 5.1 identifies the model used in the large-scale experiments as the large BART model. Section 5.3 reports that BART outperformed the best previous XSum summarization system by roughly 3.5 points on all ROUGE metrics, supporting the state-of-the-art claim. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to multitask learning, training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models (Keskar et al., 2019; ElSahar et al., 2020; Sudhakar et al., 2019; Li et al., 2020).”
What the sources say
- Supportedcites Keskar et al., 2019 source
The Introduction (p. 1) describes training CTRL on varied text sources with control codes derived from their associated structure. Language Modeling with CTRL (p. 2) identifies it as an autoregressive conditional language model, and Control Codes (p. 5) shows that the codes enable predictable variation and increasingly constrain generation. (LLM, conf=0.89)
- Partly supportedcites ElSahar et al., 2020 sourceExperimental — accuracy not yet measured
Section 4 (p. 4) says the authors enrich reviews with category-derived control codes during training. Section 7 (pp. 8–9) reports that removing the codes degrades summary quality and that generated summaries tend to follow appropriate codes. The paper supports improved control in its review-summarization model, but does not isolate heterogeneous training data as the reason for that improvement or establish the result for autoregressive language models generally. (LLM, conf=0.84)
- Partly supportedcites Sudhakar et al., 2019 sourceExperimental — accuracy not yet measured
Section 2.3.2 describes an autoregressive transformer whose B-GST input includes a special token for the target style, and Section 4.3 reports improved style-transfer results. But the paper does not show that training on heterogeneous data annotated with control codes caused that improvement; Section 2.3.3 describes reconstruction training for style transfer, not a comparison of heterogeneous control-code training against an alternative. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While our multi-target setting allows us to mitigate some of the limitations of Rouge (Conroy et al., 2011; Cohan and Goharian, 2016), we acknowledge that relying only on automated metrics is insufficient for evaluating the quality of the models.”
What the sources say
- Supportedcites Conroy et al., 2011 source
The citation is attached to the limitations of ROUGE, which the paper documents directly. In Section 3, the authors say the gap between automatic and manual evaluation persists and can force the use of manual evaluation; the Conclusion identifies linguistic quality as a deficiency of ROUGE. These passages also support the citing sentence’s broader caution about relying only on automated metrics for summary evaluation. (LLM, conf=0.91)
- Not matchedcites Cohan and Goharian, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that SCITLDR is smaller in comparison to automatically collected datasets, such as XSUM and ArXiv, but is larger in comparison to other manually collected datasets, such as SciSummNet.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this, SCITLDR contains TLDRs written from the perspective of the author ('TLDRAuth') and TLDRs written from the perspective of the peer reviewer('TLDR-PR').”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.”
- Also stated in: § 1 Introduction · p. 2 · plain
“where the latter are collected using a novel annotation protocol that produces highquality summaries while avoiding the burden of reading the full paper.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: Scaling up data collection in a specialized scientific domain is costly and challenging.
- The authors state this as an assumption: The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).
- Also stated in: § 1 Introduction · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite similarities in length, the 1-, 2-, and 3-gram mean Jaccard indices between TLDR-Auth and TLDR-PR are 15.0%, 2.5%, and 0.7%, respectively, indicating extremely little lexical overlap between the two sources of TLDRs.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most TLDRs contain between two to four nuggets (never all six), and will provide some indication of their subject area ( A ) and the paper's contributions ( C ).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that SCITLDR is more abstractive compared with other scientific domain datasets but less abstractive compared with non-scientific domain datasets.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe that SCITLDR has short summaries, like XSUM and NewsRoom, with long source documents, like BigPatent and the other scientific-domain datasets. This results in a much higher compression ratio compared with existing datasets.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TLDR-Auth tend to include results or scientific/theoretical findings ( R ) and often signal the value of their work ( V ) by describing their contributions as novel or their results as strong or state-of-the-art .”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we note that TLDR-Auth are on average 18.9 words long, while TLDR-PR are slightly longer on average at 22.9 words.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CATTS improves upon strong baselines under both automated metrics and human evaluations.”
- Also stated in: § 1 Introduction · p. 2 · boosted
“We show that CATTS applied to BART (Lewis et al., 2020), a state-of-the-art summarization model, results in performance improvement in both automated metrics and human evaluation.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 2 · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“TLDR-PR is more abstractive with a novelty score of 20.2% compared with TLDRAuth with a novelty score of 9.6%, where novelty is computed as the percentage of words in the TLDR not in the source paper.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While this up-sampling may indicate that CATTS is training on more TLDRs than BART, we allow BART training up to 20 epochs and it quickly overfits within a few epochs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average combined length of these contexts is 993 words and resulting compression ratio is 47.3, which is still higher than other datasets surveyed in Table 1.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that MatchSum has the highest extractive performance, followed by BERTSUMEXT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Curiously, despite both achieving the same Rouge-1, the generated TLDRs are quite different.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe that increasing input space from abstract-only to AIC greatly improves PACSUM 15 performance but decreases performance of both BERTSUMEXT and MatchSum.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The 'two hidden layer' detail by BART XSUM is from the paper introduction and the 'defining the appropriate sampling distributions' from CATTS XSUM is from the conclusion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For a given abstractive baseline BART or BART XSUM , our CATTS learning strategy results in improvements in both abstract-only and AIC settings.”
- Also stated in: § 6.1 Quantitative results · p. 8 · plain
“Comparing CATTS variants with their corresponding BART baselines, we observe that in the abstract-only setting, CATTS and CATTS XSUM achieve +0.5 and +1.8 Rouge-1, respectively. In the AIC setting, CATTS and CATTS XSUM achieve +2.0 and +0.9 Rouge-1, respectively.”
- Also stated in: § 8 Conclusion · p. 9 · plain
“We show that our approach improves over strong modeling baselines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.1 Quantitative results · p. 8 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our full text oracle achieves 54.5 Rouge-1, 30.6 Rouge-2, and 45.0 Rouge-L on the test set.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The average length of an abstract is 159 words and resulting compression ratio is 7.6.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Table 5, we see the systems all generate TLDRs of similar length to the average length reported in Table 1.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“PACSUM using the full text yields a Rouge-1 of 12.7, significantly worse than abstract-only.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BART XSUM and CATTS XSUM both achieve Rouge-1 of 40.7 on this paper.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Summarization in higher compression settings is challenging as it requires capturing more precisely the salient aspects of the document (Grusky et al.).”
What the sources say
- Not matchedcites Grusky et al.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Writing a TLDR of a scientific paper requires expert background knowledge and understanding of complex domain-specific language to identify the salient aspects of the paper, while maintaining faithfulness to the source and correctness of the written summary.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 1 · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually assessed every written summary, discarding TLDRs that did not adhere to the guidelines, and allowed 20/28 students who performed well to continue work beyond the first hour.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach addresses two main challenges: (1) the limited size of the training data and (2) the need for domain knowledge in order to write high-quality gold TLDRs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is not unexpected because TLDR-PR are derived from peer review comments which themselves have already gone through one stage of abstraction.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“To address the issue of multiple hypothesis testing over Rouge scores, we perform a Holm-Bonferroni (Holm, 1979) 16 correction for determining significant p -values in Table 3.”
What the sources say
- Not matchedcites Holm, 1979
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“SCITLDR is a new form of extreme summarization focused on scientific papers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised models like BERTSUMEXT and BART are trained on SCITLDR and the best model checkpoint chosen using dev set loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this setting, we have both chosen the strongest baseline and controlled for XSUM pretraining.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.
- Rests on an assumption: The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use the two-sided paired t-test against a null hypothesis of no difference to assess these differences.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since TLDRs often contain information spread across multiple sentences, we expect abstractive summarization methods to produce strong results for this task.”
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Best hyperparameters for the models are selected based on dev set Rouge-1.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BART XSUM focuses on the methodological contribution while CATTS XSUM focuses on a scientific finding.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We received responses from 29 unique authors with annotations covering 64 arXiv papers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Scaling up data collection in a specialized scientific domain is costly and challenging.
“Scaling up data collection in a specialized scientific domain is costly and challenging.”
The authors state this as an assumption.
The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).
“Assuming the typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph), domain experts can rewrite these comments into TLDRs.”
The paper relies on this without support.
Titles often contain key information about a paper.
“As titles often contain key information about a paper, we hypothesize that training a model to generate titles will allow it to learn how to locate salient information in the paper that will be also useful for generating TLDRs.”
The paper relies on this without support.
All papers have a title.
“In addition, all papers have a title, and thus we have an abundant supply of paper-title pairs for training.”
The paper relies on this without support.
Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.
“we also have human experts in computer science assess system-generated TLDRs under two criteria - informativeness and correctness.”
The paper relies on this without support.
Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.
“we ask the original authors of papers to assess the correctness of our system-generated TLDRs.”
The paper relies on this without support.
The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.
“We perform our human evaluation on BART XSUM and CATTS XSUM using the AIC input space on 51 sampled papers.”
The paper relies on this without support.
The 76 gold papers used for nugget analysis are representative.
“we perform the nuggetbased analysis for information content over systemgenerated TLDRs for the same 76 gold papers from Section 3.2.”
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 10×
“PACSUM is an extension of TextRank (Mihalcea and Tarau, 2004), utilizing directed edges and a pretrained BERT model as a sentence encoder.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“To address the issue of multiple hypothesis testing over Rouge scores, we perform a Holm-Bonferroni (Holm, 1979)16 correction for determining significant p-values in Table 3.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“The first two use BERT-base and the last one uses RoBERTa-base (Liu et al., 2019).”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 5×
“We particularly focus on BART-large, a state-of-the-art pretrained model with strong results in text generation and summarization (Lewis et al., 2019).”
BackgroundMethodInfluentialTextRank: Bringing Order into Text
Mihalcea, R. F., & Tarau, P. (2004) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 3,631
Research works printed but not linked: 3
- Paul Over. 2003. An introduction to duc 2003: Intrinsic evaluation of generic news text summarization systems. In Proceedings of Document Understanding Conference 2003 .
- In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 32693279, Hong Kong, China. Association for Computational Linguistics.
- Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018b. Syntactic scaffolds for semantic structures. In EMNLP .
Cited by
Sorted by citations
Items 101–173 of 173
Štiglic, G., Musović, K., Gosak, L., Fijačko, N., & Kocbek, P. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00118 · Cited by 4
Rajan, R. P., Jose, D. V., & Gurumoorthy, R. (2023) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2023.06.05 · Cited by 3
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Wadden, D., Shi, K., Morrison, J., Li, A., Naik, A., Singh, S., Barzilay, N., Lo, K., Hope, T., Soldaini, L., et al. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.310 · Cited by 3
Self-citationArévalo-Royo, J., Montalvo, F. J. F., Latorre‐Biel, J., Macías, E. J., Martínez, E., & Blanco‐Fernández, J. (2025) · Applied Sciences · doi:10.3390/app152010913 · Cited by 3
CocoSciSum: A Scientific Summarization Toolkit with Compositional Controllability
Ding, Y., Qin, Y., Liu, Q., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.47 · Cited by 2
“Inspired by (Cachola et al., 2020), we shuffle two finetuning datasets and then train the model with the shuffled dataset to generalize on both attributes.”
MethodComparative Analysis of Pretrained Encoder-Decoder Transformer Models for Extreme Text Summarization
RajyaLakshmi, T., & Kuppusamy, K. S. (2023) · International Conference on Advances in Computational Intelligence and Communication (ICACIC) · doi:10.1109/icacic59454.2023.10435363 · Cited by 3
Automatic Text Summarization-Based Transformers Architecture
Al-Banna, A. A., & Al-Mashhadany, A. K. (2023) · International Conference on Information Technology, Applied Mathematics and Statistics (ICITAMS) · doi:10.1109/icitams57610.2023.10525560 · Cited by 2
“This allows for more parallelization than RNNs, resulting in shorter training durations [27]; Figure 1 shows the difference between traditional RNNs that process embedding sequentially and Transformers.”
MethodA Comprehensive Study of Automatic Text Summarization Techniques
Vaishali, Sehgal, G., & Dixit, P. (2024) · International Conference on Emerging Innovations and Advanced Computing (INNOCOMP) · doi:10.1109/innocomp63224.2024.00119 · Cited by 2
Kopitar, L., Kocbek, P., Gosak, L., & Štiglic, G. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-13619-1.00007-6 · Cited by 3
Deep Learning-Based Scientific Document Summarization Considering Citation
Jyoti, D., Mahato, D. P., & Srivastava, J. (2025) · SN Computer Science · doi:10.1007/s42979-025-03821-9 · Cited by 3
“Studies such as TLDR [28] and SCITLDR [29] focus on extreme summarization, emphasizing brevity and precision.”
Kocbek, P., Gosak, L., Musović, K., & Štiglic, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09342-5_37 · Cited by 2
Summarization of Unstructured Medical Data for Accurate Medical Prognosis—A Learning Approach
Mishra, A., & Soni, S. (2023) · Cognitive science and technology · doi:10.1007/978-981-19-8086-2_79 · Cited by 2
Cross-lingual Science Journalism: Select, Simplify and Rewrite Summaries for Non-expert Readers
Fatima, M., & Strube, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.103 · Cited by 2
“…ArXiv, PubMed and Medline (Kim et al., 2016a; Nikolov et al., 2018; Cohan et al., 2018), Science Daily (Dan-govski et al., 2019), the ACL anthology network (Yasunaga et al., 2019), scientific blogs (Vada-1844 palli et al., 2018b,a), BBC (Narayan et al., 2018) and Open Review (Cachola et al., 2020).”
BackgroundInfluentialAutomatic Text Summarization Based on Pre-trained Models
Al-Banna, A. A., & Al-Mashhadany, A. K. (2023) · doi:10.1109/aiccit57614.2023.10218006 · Cited by 2
“The validation datasets are made up of 619 source and target pairings, whereas the test dataset is made up of 619 samples [21].”
BackgroundThe current state of summarization
Retkowski, F. (2023) · AI critique · doi:10.1515/9783839467664-016 · Cited by 6
Towards Generating Citation Sentences for Multiple References with Intent ControlPreprint
Wu, J.-Y., Shieh, A. T.-W., Hsu, S.-J., & Chen, Y.-N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01332 · Cited by 9
“The recent study by Cachola et al. (2020) introduced the SciTLDR dataset.”
Enhancing Literature Review with LLM and NLP Methods. Algorithmic Trading Case.Preprint
Łaniewski, S., & Ślepaczuk, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4996945 · Cited by 4
A Multidisciplinary Multimodal Aligned Dataset for Academic Data Processing
Song, H., Xu, H., Wang, Z., Wang, Y., & Li, J. (2025) · Scientific Data · doi:10.1038/s41597-025-04415-z · Cited by 6
Cross-Domain Document Summarization Model via Two-Stage Curriculum Learning
Lee, S., Kim, G., & Kang, S. (2024) · Electronics · doi:10.3390/electronics13173425 · Cited by 1
Cross-Domain Robustness of Transformer-Based Keyphrase Generation
Glazkova, A. V., & Morozov, D. A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67826-4_19 · Cited by 2
“In particular, abstractive summarization models were applied for generating summaries in news [5,13,20,52], scientific [6,35,45], sport [31], and financial domains [42,51].”
MethodCase Study of AI Application in Scholarly Communication—ScienceON
Yoo, S., Seon, C., & Lee, T. (2025) · Library trends · doi:10.1353/lib.2025.a961192 · Cited by 2
Foundation Models for Text Generation
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_6 · Cited by 3
Twist Decoding: Diverse Generators Guide Each Other
Kasai, J., Sakaguchi, K., Bras, R. J. L., Peng, H., Lu, X., Radev, D., Choi, Y., & Smith, N. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.326 · Cited by 4
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 5
Fatima, M., Kolber, T., Markert, K., et al. (2023) · doi:10.18653/v1/2023.newsum-1.3 · Cited by 1
Linguistic Compression in Single-Sentence Human-Written Summaries
Yin, F., & van Schijndel, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.532 · Cited by 1
“We selected three popular datasets for extreme automatic summarization: WikiHow (Koupaee and Wang, 2018), XSum (Narayan et al., 2018), and Sc-iTLDR (Cachola et al., 2020).”
MethodRoy, S. S., & Mercer, R. E. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.8 · Cited by 1
NLP Applications—Biomedical Literature
Zhang, R., Kastrin, A., Hristovski, D., Marcelo Fiszman, B., & Halil Kilicoglu, B. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_13 · Cited by 1
Improving Abstractive Summarization with Unsupervised Dynamic LoRA Mixtures
Chernyshev, D. (2024) · Lobachevskii Journal of Mathematics · doi:10.1134/s1995080224603710 · Cited by 1
A Directory of Datasets for Mining Software Repositories
Diamantopoulos, T., & Symeonidis, A. L. (2025) · Data · doi:10.3390/data10030028 · Cited by 2
Zhou, Y., Wei, J., Sun, Y., & Du, W. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00004-7 · Cited by 3
“Due to this constraint, they often analyze truncated portions of a document, leading to potential loss of pivotal information (Cachola et al. 2020).”
A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
Yamamoto, S., Suzuki, R., Fukusato, T., Kataoka, H., & Morishima, S. (2021) · Applied Sciences · doi:10.3390/app11146287 · Cited by 1
“For example, Semantic Scholar (https://www. semanticscholar.org/ last accessed on 9 June 2021), an academic search engine, shows single-sentence paper summaries generated by an automatic summarization method [32] to help users judge which papers are relevant.”
Wang, C. (2022) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/icces54183.2022.9835878 · Cited by 1
Overview of NLPCC Shared Task 2: Multi-perspective Scientific Machine Reading Comprehension
Zhang, X., Zheng, H., Nie, Y., & Mao, X.-L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_10 · Cited by 1
D2S: Document-to-Slide Generation Via Query-Based Text SummarizationPreprint
Sun, E., Hou, Y., Wang, D., Zhang, Y., & Wang, N. X. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03664 · Cited by 1
“Although there has been increasing interest in constructing large-scale single-document text summarization corpora (CNN/DM (Hermann et al., 2015), Newsroom (Grusky et al., 2018), XSum (Narayan et al., 2018), TLDR (Cachola et al., 2020)) and developing various approaches to address this task (Pointer Generator (See et…”
BackgroundMethodBringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific DocumentsPreprint
Meng, R., Thaker, K. M., Zhang, L., Dong, Y., Yuan, X., Wang, T., & He, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00130 · Cited by 1
“, 2020), which has recently achieved SOTA performance on abstractive summarization tasks with scientific articles (Cachola et al., 2020).”
BackgroundMethodInfluentialPortenoy, J., Radensky, M., West, J. D., et al. (2021) · arXiv (Cornell University) · Cited by 1
Self-citationLee, S., Yang, K., Park, C., Sedoc, J., & Lim, H. (2021) · arXiv (Cornell University) · Cited by 1
Aspect-based Document Similarity for Literature Recommender Systems
Ostendorff, M. (2023) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9937 · Cited by 1
Abstractive Summarization Model for Summarizing Scientific ArticlePreprint
Ülker, M., & Özer, A. B. (2024) · Preprints.org · doi:10.20944/preprints202405.1123.v1 · Cited by 2
Dube, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-68624-6_10
VerbCL: A Dataset of Verbatim Quotes for Highlight Extraction in Case Law
Rossi, J., Vakulenko, S., & Kanoulas, E. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482021
Paperswithtopic: Topic Identification from Paper Title Only
Cho, D., & Wallraven, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-02444-3_19
Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLPPreprint
Jin, Z., von Kügelgen, J., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03618
“For example, summarization data collected from the TL;DR of scientific papers SciTldr (Cachola et al., 2020) should be causal, as the TL;DR summaries on OpenReview (some from authors when submitting the paper, others derived from the beginning of peer reviews) were likely composed after the original papers or reviews…”
MethodReal or Fake: An intrinsic analysis using supervised machine learning algorithms
Biwalkar, A., Rao, A., & Shah, K. (2021) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac52330.2021.9640675
AUTOMATSKA SUMARIZACIJA VESTI O KRIPTOVALUTAMA
Ogrizović, U. (2022) · Zbornik radova Fakulteta tehničkih nauka u Novom Sadu · doi:10.24867/16be42ogrizovic
Detecting Uninformative Research Job Titles via Classifier Failures—Zero Shot Approach
Víta, M. (2022) · Annals of Computer Science and Information Systems · doi:10.15439/2022f285
Position Papers of the 17th Conference on Computer Science and Intelligence Systems
(2022) · Annals of Computer Science and Information Systems · doi:10.15439/978-83-965897-2-9 · Cited by 1
Jost, J., & Restrepo, G. (2022) · Wissenschaft und Philosophie · doi:10.1007/978-3-031-10094-9_6
Chaining and the temporal dynamics of scientists’ publishing behaviour
Liu, E., & Xu, Y. (2022) · PLoS ONE · doi:10.1371/journal.pone.0278389
Jung, J., Lee, J., Choi, S., & Baek, W. (2022) · Energies · doi:10.3390/en16010033
Approaching history of chemistry through big data on chemical reactions and compounds
Restrepo, G. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85713-0.00033-5
Kurian, K. S., & Mathew, S. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-8479-4_13
VADIS – A Variable Detection, Interlinking and Summarization System
Kartal, Y. S., Shahid, M. A., Takeshita, S., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_22
“For summarization, there is a line of work focusing on scientific document summarization from domain-specific datasets [3] to the models that exploit the unique properties such as citation contexts [5] or discourse structures [4,1].”
BackgroundTEXT SUMMARIZATION USING DEEP LEARNING MODEL
Prajapati, M., & Rai, V. (2024) · doi:10.55524/csistw.2024.12.1.73 · Cited by 1
Research Paper Summarization Using Extractive Approach
Pawade, D., M.Sakhapara, A., Thakare, S., et al. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-3242-5_17
Leveraging Automated POS Tagging to Decode Parent-Infant Interactions in Digital Gameplays
Yeh, W.-T., Knott, A., & Henderson, A. M. E. (2024) · International Conference on Innovation in Artificial Intelligence (ICIAI) · doi:10.1145/3655497.3655528
“However, most of the research has been done for single-speaker documents, such as news articles [13-14], scientific documents [15] and strictly structured formats [16].”
Enhancing Chemistry-Domain Scientific Paper Summarization by Knowledge Graphs
Qu, Y., Yang, J. T., Chen, W., Jiao, Y., Yang, L., & Ma, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0814-0_21
Leveraging Pre-Trained Models for Abstractive Summarization of Computer Science Research Papers
Pepita, P., Hendra, J. S., Wulandhari, L. A., et al. (2025) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict65714.2025.11100819
“The dataset was obtained through the official repository maintained by the MetaText project [24].”
Health applications of shaped charge learning
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30046-2.00006-5
Kartal, Y. S., Schott, K., & Kern, D. (2025) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3720553.3746670 · Cited by 1
“Scientific document summarization models aim to condense entire documents or abstracts into concise summaries [11, 30, 39].”
Advancing text summarization with specialized datasets: computer science and geography domains
Faheem, H. B., Wali, A., Muzammil, S., Mahmood, H., Sarfraz, S., & Asif, H. (2025) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf124
Read Between the Tokens: Differentiable Text Pruning via Perturbed Top-k Selection
Italiani, P., Ragazzi, L., & Moro, G. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3629289
Refinement-Guided Critique Learning: A Framework for Training Critique Models
Xiang, C. J., Zheng, J., Mu, X., Yu, T., Zhang, L., Sun, C., Shi, L., Wang, F., & Yang, M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.104002
Tag augmentation and auto recommendation on social media: a hybrid optimization framework
Saibaba, C. M. H., Krishnan, S. B., Shankar, S. S., Chakrabarti, P., Chakrabarti, T., & Margala, M. (2025) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-025-03105-8
LLM-Based Abstractive Summarization of PubMed Articles
Ho, E., Heidari, M., Zad, S., Boroojeni, K. G., & Headden, C. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0433-6_14
Enhancing Research Paper Title Generation Using GPT-2 for Natural Language Processing
Rokade, S., Biradar, R., Patil, K., et al. (2025) · doi:10.1109/iconat66879.2025.11362623
Ramillano, A. R. B., & Recario, R. N. C. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11467566
“Corpora such as The Nordic Pile [26], SciTLDR [27], GovReport [28], and BillSum [29] support summarization research, with evaluation commonly based on ROUGE metrics [30].”
Enhancing online reading experiences for scientific documents: a study on layout usability
Saeed, Z., Faisal, C. M. N., Aunsa, A. D. H., & Iqbal, M. Z. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2671392
Mix and Match: Context Pairing for Scalable Topic-Controlled Educational Summarisation
Yodthap, N., Intharah, T., & Bulathwela, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_20
Intelligent Analysis Framework for Academic Papers with Local LLM Integration
Xiaoye, H., & Wu, Y. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-30836-8_13
Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Garg, K., Shaik, F., Bandyopadhyay, S., & Caragea, C. (2025) · arXiv · doi:10.48550/arxiv.2508.14273
Schott, K., Silva, K., Frommholz, I., Mayr, P., Kern, D., & Hienert, D. (2026) · arXiv · doi:10.48550/arxiv.2607.03421
Metadata
- Authors
- Cachola, I., Lo, K., Cohan, A., & Weld, D.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Published
- 2020
- Pages
- 4766-4777
- OpenAlex
- W3021892802
- Semantic Scholar
- 3502a542b2e98d9094e1880a30f652d4170b9534
- Cited by
- 169
- References
- 51
- Citations by year
- 2026: 6 · 2025: 18 · 2024: 26 · 2023: 48 · 2022: 35 · 2021: 34 · 2020: 2
- FWCI
- 11.35
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1936940, 2040196)
- University of Washington
- Office of Naval Research (N00014- 18-1-2193, N00014-18, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- annotation protocol, computer science literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
