CoChat
Sign Up
Cited by 171Advanced Text Analysis TechniquesTopic ModelingText Readability and Simplification

TLDR: Extreme Summarization of Scientific Documents

Open access
Cachola, Isabel, Lo, Kyle, Cohan, Arman, Weld, Daniel·2020·Findings of the Association for Computational Linguistics: EMNLP 2020
DOI 10.18653/v1/2020.findings-emnlp.428
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce TLDR generation, a new form of extreme summarization, for scientific papers.TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.To facilitate study on this task, we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers.SCITLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal.CATTS improves upon strong baselines under both automated metrics and human evaluations.Data and code are publicly available at https://github.com/allenai/scitldr. Dataset constructionOverview We introduce SCITLDR, a new multitarget dataset of 5,411 TLDRs over 3,229 scientific papers in the computer science domain.3 The training set contains 1,992 papers, each with a single gold TLDR.The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.This is unlike the majority of existing 3 See Appendix Table 9 for full venue breakdown.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces TLDR generation for scientific papers and releases SciTLDR, a multi-target dataset of 5,411 TLDRs over 3,229 computer science papers. It proposes CATTS, which uses paper titles as an auxiliary training signal, and reports improvements over strong extractive and abstractive baselines on automated and human evaluations.

Key findings

  • SciTLDR contains 5,411 TLDRs over 3,229 papers: 1,992 training papers with one gold TLDR each, 619 dev papers with 1,452 TLDRs, and 618 test papers with 1,967 TLDRs. Its summaries average 21 words, with a compression ratio of 238.1 and 15.2% novel words.
  • Author-written TLDRs average 18.9 words and peer-review-derived TLDRs average 22.9 words; lexical overlap is very low (1-, 2-, and 3-gram mean Jaccard indices of 15.0%, 2.5%, and 0.7%). TLDR-PR is more abstractive, with 20.2% novel words versus 9.6% for TLDR-Auth.
  • CATTS improves over BART baselines: in abstract-only input, CATTS gains +0.5 Rouge-1 and CATTS_XSUM gains +1.8 Rouge-1; in AIC input, CATTS gains +2.0 Rouge-1 and CATTS_XSUM gains +0.9 Rouge-1. In AIC, CATTS reaches 44.9 Rouge-1, 22.6 Rouge-2, and 37.3 Rouge-L versus BART’s 42.9/20.8/35.1.
  • Human evaluation found CATTS_XSUM more informative than BART_XSUM (MRR 0.54 vs 0.42), comparable to gold TLDR-Auth (0.53) but below TLDR-PR (0.60). Correctness did not differ: both models averaged 2.5, with 42 ties, 10 cases favoring BART_XSUM, and 12 favoring CATTS_XSUM.

Methods

  • Dataset construction combined author-written TLDRs collected from OpenReview with expert-derived TLDR-PR summaries created by rewriting the first 128 words of peer review comments. Twenty-eight University of Washington undergraduate computer science students were recruited, 20 continued after initial training, and annotators were paid $20/hour; full texts were parsed with S2ORC.
  • Baselines included extractive PACSUM, BERTSUMEXT, and MatchSum, plus abstractive BART-large and BART-large finetuned on XSUM. CATTS appends control codes for TLDR and title generation, trains with an additional 20K arXiv paper-title pairs, and upsamples TLDR instances; experiments use abstract-only and abstract/introduction/conclusion (AIC) input spaces.
  • Evaluation uses max Rouge over multiple gold TLDRs per paper, plus human evaluation: nugget-based informativeness on 76 sampled gold papers and author-rated correctness from 29 unique authors covering 64 arXiv papers.

Limitations

  • Stated in the paper: experiments are limited to abstract-only and AIC input spaces, though full texts are provided for future work; the idea of a TLDR may differ across academic disciplines and is left for future work; a small fraction (<5%) of papers lacked available PDFs and were included using abstract-only text.
  • Evident from the text: the dataset is restricted to computer science papers, and the human correctness evaluation is small (29 authors, 64 papers) and found no difference between BART_XSUM and CATTS_XSUM.
  • Evident from the text: informativeness analysis uses only 76 sampled gold papers, and automated evaluation relies on max Rouge over multiple gold TLDRs, which rewards matching any gold summary and may not fully capture factual correctness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2020.findings-emnlp.428

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 21 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 2 Dataset construction · p. 2

    Variability in human-written summaries can negatively impact the reliability of automated summarization metrics like Rouge.

    “As evidenced by earlier work in summarization evaluation (Cohan and Goharian, 2016), variability in human-written summaries (Zechner, 1996; Harman and Over, 2004) can negatively impact the reliability of automated summarization metrics like Rouge (Lin, 2004).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Cohan and Goharian, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zechner, 1996 source
    • cites Lin, 2004

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation
    § 4 CATTS · p. 5

    Control codes can be used to effectively incorporate scaffold tasks (e.g. title generation) for denoising autoencoders like BART.

    “We demonstrate that control codes can be used to effectively incorporate scaffold tasks (e.g. title generation) for denoising autoencoders like BART (Lewis et al., 2020).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Lewis et al., 2020 source
  • Findingown data
    § 2 Dataset construction · p. 2

    SCITLDR is unlike the majority of existing summarization datasets that assume only one gold summary for a given document.

    “This is unlike the majority of existing summarization datasets that assume only one gold summary for a given document.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 3.3 Variability in TLDRs · p. 4

    TLDR-Auth and TLDR-PR can differ greatly, even when they contain the same information content.

    “We can also observe through qualitative examples in Figure 3 how TLDR-Auth and TLDR-PR can differ greatly, even when they contain the same information content.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 3.2 Information content · p. 4

    TLDR-PR focus more on articulating problems the paper addresses (P).

    “In contrast, TLDRPR focus more on articulating problems the paper addresses ( P ).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 28

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    Existing work in summarization of scientific documents aims to generate abstracts or provide complimentary summaries to abstracts.

    “The goal of existing work in summarization of scientific documents is to generate abstracts or provide complimentary summaries to abstracts. (Collins et al., 2017; Cohan et al., 2018; Chandrasekaran et al., 2019; Yasunaga et al., 2019).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Collins et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Cohan et al., 2018 source
    • cites Chandrasekaran et al., 2019 source

      The Introduction (p. 1) describes citation-based mini-summaries, and the Task section (p. 2) asks systems to generate a structured summary of a scientific paper. It lists abstracts alongside community and human-written summaries, but does not say that the task aims to generate abstracts or that its summaries are intended to complement abstracts. It also does not characterize the goals of existing work generally. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4 CATTS · p. 5

    Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification.

    “Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification (Swayamdipta et al., 2018b; Cohan et al., 2019).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Swayamdipta et al., 2018b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Cohan et al., 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.1 Baselines · p. 5

    BART-large achieves state-of-the-art results in summarization on XSUM.

    “We use BART-large, which achieves state-of-the-art results in summarization on XSUM.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lewis et al., 2020 source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 4 CATTS · p. 5

    Training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models.

    “Similar to multitask learning, training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models (Keskar et al., 2019; ElSahar et al., 2020; Sudhakar et al., 2019; Li et al., 2020).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Keskar et al., 2019 source
    • cites ElSahar et al., 2020 source

      Section 4 (p. 4) says the authors enrich reviews with category-derived control codes during training. Section 7 (pp. 8–9) reports that removing the codes degrades summary quality and that generated summaries tend to follow appropriate codes. The paper supports improved control in its review-summarization model, but does not isolate heterogeneous training data as the reason for that improvement or establish the result for autoregressive language models generally. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Sudhakar et al., 2019 source

      Section 2.3.2 describes an autoregressive transformer whose B-GST input includes a special token for the target style, and Section 4.3 reports improved style-transfer results. But the paper does not show that training on heterogeneous data annotated with control codes caused that improvement; Section 2.3.3 describes reconstruction training for style transfer, not a comparison of heterogeneous control-code training against an alternative. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation
    § 5.4 Evaluation · p. 7

    Relying only on automated metrics is insufficient for evaluating the quality of the models.

    “While our multi-target setting allows us to mitigate some of the limitations of Rouge (Conroy et al., 2011; Cohan and Goharian, 2016), we acknowledge that relying only on automated metrics is insufficient for evaluating the quality of the models.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Conroy et al., 2011 source
    • cites Cohan and Goharian, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 3.1 Compression and abstractiveness · p. 4

    SCITLDR is smaller in comparison to automatically collected datasets, such as XSUM and ArXiv, but is larger in comparison to other manually collected datasets, such as SciSummNet.

    “We also observe that SCITLDR is smaller in comparison to automatically collected datasets, such as XSUM and ArXiv, but is larger in comparison to other manually collected datasets, such as SciSummNet.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 2 Dataset construction · p. 2

    SCITLDR contains TLDRs written from the perspective of the author (TLDR-Auth) and TLDRs written from the perspective of the peer reviewer (TLDR-PR).

    “To address this, SCITLDR contains TLDRs written from the perspective of the author ('TLDRAuth') and TLDRs written from the perspective of the peer reviewer('TLDR-PR').”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § Abstract · p. 1

    The expert-derived TLDRs are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.

    “where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “where the latter are collected using a novel annotation protocol that produces highquality summaries while avoiding the burden of reading the full paper.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Scaling up data collection in a specialized scientific domain is costly and challenging.
    • The authors state this as an assumption: The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).

  • May combine two claims.

    Findingown data
    § 3.3 Variability in TLDRs · p. 4

    The 1-, 2-, and 3-gram mean Jaccard indices between TLDR-Auth and TLDR-PR are 15.0%, 2.5%, and 0.7%, respectively, indicating extremely little lexical overlap between the two sources of TLDRs.

    “Despite similarities in length, the 1-, 2-, and 3-gram mean Jaccard indices between TLDR-Auth and TLDR-PR are 15.0%, 2.5%, and 0.7%, respectively, indicating extremely little lexical overlap between the two sources of TLDRs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2 Dataset construction · p. 2

    SCITLDR's dev and test sets contain 619 and 618 papers, with 1,452 and 1,967 TLDRs, respectively.

    “The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Information content · p. 4

    Most TLDRs contain between two to four nuggets (never all six), and will provide some indication of their subject area (A) and the paper's contributions (C).

    “Most TLDRs contain between two to four nuggets (never all six), and will provide some indication of their subject area ( A ) and the paper's contributions ( C ).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.1 Compression and abstractiveness · p. 4

    SCITLDR is more abstractive compared with other scientific domain datasets but less abstractive compared with non-scientific domain datasets.

    “We observe that SCITLDR is more abstractive compared with other scientific domain datasets but less abstractive compared with non-scientific domain datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 3.1 Compression and abstractiveness · p. 3

    SCITLDR has short summaries, like XSUM and NewsRoom, with long source documents, like BigPatent and the other scientific-domain datasets, resulting in a much higher compression ratio compared with existing datasets.

    “We observe that SCITLDR has short summaries, like XSUM and NewsRoom, with long source documents, like BigPatent and the other scientific-domain datasets. This results in a much higher compression ratio compared with existing datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Information content · p. 4

    TLDR-Auth tend to include results or scientific/theoretical findings (R) and often signal the value of their work (V) by describing their contributions as novel or their results as strong or state-of-the-art.

    “TLDR-Auth tend to include results or scientific/theoretical findings ( R ) and often signal the value of their work ( V ) by describing their contributions as novel or their results as strong or state-of-the-art .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.3 Variability in TLDRs · p. 4

    TLDR-Auth are on average 18.9 words long, while TLDR-PR are slightly longer on average at 22.9 words.

    “First, we note that TLDR-Auth are on average 18.9 words long, while TLDR-PR are slightly longer on average at 22.9 words.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    CATTS improves upon strong baselines under both automated metrics and human evaluations.

    “CATTS improves upon strong baselines under both automated metrics and human evaluations.”
    • Also stated in: § 1 Introduction · p. 2 · boosted
      “We show that CATTS applied to BART (Lewis et al., 2020), a state-of-the-art summarization model, results in performance improvement in both automated metrics and human evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 3.3 Variability in TLDRs · p. 4

    TLDR-PR is more abstractive with a novelty score of 20.2% compared with TLDR-Auth with a novelty score of 9.6%, where novelty is computed as the percentage of words in the TLDR not in the source paper.

    “TLDR-PR is more abstractive with a novelty score of 20.2% compared with TLDRAuth with a novelty score of 9.6%, where novelty is computed as the percentage of words in the TLDR not in the source paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 5.4 Evaluation · p. 6

    BART training is allowed up to 20 epochs and it quickly overfits within a few epochs.

    “While this up-sampling may indicate that CATTS is training on more TLDRs than BART, we allow BART training up to 20 epochs and it quickly overfits within a few epochs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Input space · p. 6

    The average combined length of AIC contexts is 993 words and resulting compression ratio is 47.3, which is still higher than other datasets surveyed in Table 1.

    “The average combined length of these contexts is 993 words and resulting compression ratio is 47.3, which is still higher than other datasets surveyed in Table 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Quantitative results · p. 7

    MatchSum has the highest extractive performance, followed by BERTSUMEXT.

    “We observe that MatchSum has the highest extractive performance, followed by BERTSUMEXT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Analysis · p. 8

    Despite both achieving the same Rouge-1, the generated TLDRs by BART_XSUM and CATTS_XSUM are quite different.

    “Curiously, despite both achieving the same Rouge-1, the generated TLDRs are quite different.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 6.1 Quantitative results · p. 7

    Increasing input space from abstract-only to AIC decreases performance of both BERTSUMEXT and MatchSum.

    “We observe that increasing input space from abstract-only to AIC greatly improves PACSUM 15 performance but decreases performance of both BERTSUMEXT and MatchSum.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 6.3 Analysis · p. 8

    The 'two hidden layer' detail by BART_XSUM is from the paper introduction and the 'defining the appropriate sampling distributions' from CATTS_XSUM is from the conclusion.

    “The 'two hidden layer' detail by BART XSUM is from the paper introduction and the 'defining the appropriate sampling distributions' from CATTS XSUM is from the conclusion.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Quantitative results · p. 8

    CATTS learning strategy improves Rouge-1 over corresponding BART baselines in both abstract-only and AIC settings.

    “For a given abstractive baseline BART or BART XSUM , our CATTS learning strategy results in improvements in both abstract-only and AIC settings.”
    • Also stated in: § 6.1 Quantitative results · p. 8 · plain
      “Comparing CATTS variants with their corresponding BART baselines, we observe that in the abstract-only setting, CATTS and CATTS XSUM achieve +0.5 and +1.8 Rouge-1, respectively. In the AIC setting, CATTS and CATTS XSUM achieve +2.0 and +0.9 Rouge-1, respectively.”
    • Also stated in: § 8 Conclusion · p. 9 · plain
      “We show that our approach improves over strong modeling baselines.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Baselines · p. 6

    The full text oracle achieves 54.5 Rouge-1, 30.6 Rouge-2, and 45.0 Rouge-L on the test set.

    “Our full text oracle achieves 54.5 Rouge-1, 30.6 Rouge-2, and 45.0 Rouge-L on the test set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § 5.2 Input space · p. 6

    The average length of an abstract is 159 words and resulting compression ratio is 7.6.

    “The average length of an abstract is 159 words and resulting compression ratio is 7.6.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Analysis · p. 8

    The systems all generate TLDRs of similar length to the average length reported in Table 1.

    “From Table 5, we see the systems all generate TLDRs of similar length to the average length reported in Table 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 6.1 Quantitative results · p. 7

    PACSUM using the full text yields a Rouge-1 of 12.7, significantly worse than abstract-only.

    “PACSUM using the full text yields a Rouge-1 of 12.7, significantly worse than abstract-only.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 7 Related work · p. 9

    BART_XSUM and CATTS_XSUM both achieve Rouge-1 of 40.7 on the example paper.

    “BART XSUM and CATTS XSUM both achieve Rouge-1 of 40.7 on this paper.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.1 Compression and abstractiveness · p. 3

    Summarization in higher compression settings is challenging as it requires capturing more precisely the salient aspects of the document.

    “Summarization in higher compression settings is challenging as it requires capturing more precisely the salient aspects of the document (Grusky et al.).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Grusky et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § Abstract · p. 1

    TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.

    “TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.”
    • Also stated in: § 1 Introduction · p. 1 · plain
      “Writing a TLDR of a scientific paper requires expert background knowledge and understanding of complex domain-specific language to identify the salient aspects of the paper, while maintaining faithfulness to the source and correctness of the written summary.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § Rewriting peer reviews into TLDR-PR pairs · p. 3

    The authors manually assessed every written summary, discarding TLDRs that did not adhere to the guidelines, and allowed 20/28 students who performed well to continue work beyond the first hour.

    “We manually assessed every written summary, discarding TLDRs that did not adhere to the guidelines, and allowed 20/28 students who performed well to continue work beyond the first hour.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 4 CATTS · p. 5

    CATTS addresses two main challenges: the limited size of the training data and the need for domain knowledge in order to write high-quality gold TLDRs.

    “Our approach addresses two main challenges: (1) the limited size of the training data and (2) the need for domain knowledge in order to write high-quality gold TLDRs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.3 Variability in TLDRs · p. 4

    The higher abstractiveness of TLDR-PR is not unexpected because TLDR-PR are derived from peer review comments which themselves have already gone through one stage of abstraction.

    “This is not unexpected because TLDR-PR are derived from peer review comments which themselves have already gone through one stage of abstraction.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 8 Conclusion · p. 9

    The multiple target summaries in SCITLDR can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.

    “Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 6.1 Quantitative results · p. 8

    The study performed a Holm-Bonferroni correction for multiple hypothesis testing over Rouge scores.

    “To address the issue of multiple hypothesis testing over Rouge scores, we perform a Holm-Bonferroni (Holm, 1979) 16 correction for determining significant p -values in Table 3.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Holm, 1979

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 7 Related work · p. 9

    SCITLDR is a new form of extreme summarization focused on scientific papers.

    “SCITLDR is a new form of extreme summarization focused on scientific papers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 5.3 Training and implementation details · p. 6

    Supervised models like BERTSUMEXT and BART are trained on SCITLDR and the best model checkpoint chosen using dev set loss.

    “Supervised models like BERTSUMEXT and BART are trained on SCITLDR and the best model checkpoint chosen using dev set loss.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 8 Conclusion · p. 9

    The experiments are limited to abstract-only and AIC input spaces.

    “Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 6.2 Human evaluation · p. 8

    In the human evaluation, the authors chose the strongest baseline and controlled for XSUM pretraining.

    “In this setting, we have both chosen the strongest baseline and controlled for XSUM pretraining.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.
    • Rests on an assumption: The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 6.1 Quantitative results · p. 8

    The study used a two-sided paired t-test against a null hypothesis of no difference to assess differences.

    “We use the two-sided paired t-test against a null hypothesis of no difference to assess these differences.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 5.1 Baselines · p. 5

    Since TLDRs often contain information spread across multiple sentences, abstractive summarization methods are expected to produce strong results for this task.

    “Since TLDRs often contain information spread across multiple sentences, we expect abstractive summarization methods to produce strong results for this task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § 8 Conclusion · p. 9

    The multiple target summaries in SCITLDR reflect diverse perspectives.

    “Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5.3 Training and implementation details · p. 6

    Best hyperparameters for the models are selected based on dev set Rouge-1.

    “Best hyperparameters for the models are selected based on dev set Rouge-1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § 6.3 Analysis · p. 8

    BART_XSUM focuses on the methodological contribution while CATTS_XSUM focuses on a scientific finding.

    “BART XSUM focuses on the methodological contribution while CATTS XSUM focuses on a scientific finding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § 5.4 Evaluation · p. 7

    The human evaluation received responses from 29 unique authors with annotations covering 64 arXiv papers.

    “We received responses from 29 unique authors with annotations covering 64 arXiv papers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 8 Conclusion · p. 9

    The full text of source papers is provided to support research into using longer input contexts.

    “Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
    Quote in the paperNot a substantial claim
47 of 47

  • The paper relies on this without support.

    Scaling up data collection in a specialized scientific domain is costly and challenging.

    “Scaling up data collection in a specialized scientific domain is costly and challenging.”

  • The authors state this as an assumption.

    The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).

    “Assuming the typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph), domain experts can rewrite these comments into TLDRs.”

  • The paper relies on this without support.

    Titles often contain key information about a paper.

    “As titles often contain key information about a paper, we hypothesize that training a model to generate titles will allow it to learn how to locate salient information in the paper that will be also useful for generating TLDRs.”

  • The paper relies on this without support.

    All papers have a title.

    “In addition, all papers have a title, and thus we have an abundant supply of paper-title pairs for training.”

  • The paper relies on this without support.

    Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.

    “we also have human experts in computer science assess system-generated TLDRs under two criteria - informativeness and correctness.”

  • The paper relies on this without support.

    Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.

    “we ask the original authors of papers to assess the correctness of our system-generated TLDRs.”

  • The paper relies on this without support.

    The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.

    “We perform our human evaluation on BART XSUM and CATTS XSUM using the AIC input space on 51 sampled papers.”

  • The paper relies on this without support.

    The 76 gold papers used for nugget analysis are representative.

    “we perform the nuggetbased analysis for information content over systemgenerated TLDRs for the same 76 gold papers from Section 3.2.”
8 of 8

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Paul Over. 2003. An introduction to duc 2003: Intrinsic evaluation of generic news text summarization systems. In Proceedings of Document Understanding Conference 2003 .
  2. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 32693279, Hong Kong, China. Association for Computational Linguistics.
  3. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018b. Syntactic scaffolds for semantic structures. In EMNLP .

Cited by

Sorted by citations

Related

Authors
Cachola, I., Lo, K., Cohan, A., & Weld, D.
Affiliations
  • Allen Institute for Artificial Intelligence (US)
  • University of Washington (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2020
Publisher
Association for Computational Linguistics
Published
2020
Pages
4766-4777
DOI
10.18653/v1/2020.findings-emnlp.428
OpenAlex
W3021892802
Semantic Scholar
3502a542b2e98d9094e1880a30f652d4170b9534
Cited by
169
References
51
Citations by year
2026: 6 · 2025: 18 · 2024: 26 · 2023: 48 · 2022: 35 · 2021: 34 · 2020: 2
FWCI
11.35
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1936940, 2040196)
  • University of Washington
  • Office of Naval Research (N00014- 18-1-2193, N00014-18, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
annotation protocol, computer science literature

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-21.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-21.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review