The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 10 of the 21 sources its claims cite were checked for support.
TLDR: Extreme Summarization of Scientific Documents
Open accessAbstract
We introduce TLDR generation, a new form of extreme summarization, for scientific papers.TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.To facilitate study on this task, we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers.SCITLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal.CATTS improves upon strong baselines under both automated metrics and human evaluations.Data and code are publicly available at https://github.com/allenai/scitldr. Dataset constructionOverview We introduce SCITLDR, a new multitarget dataset of 5,411 TLDRs over 3,229 scientific papers in the computer science domain.3 The training set contains 1,992 papers, each with a single gold TLDR.The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.This is unlike the majority of existing 3 See Appendix Table 9 for full venue breakdown.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces TLDR generation for scientific papers and releases SciTLDR, a multi-target dataset of 5,411 TLDRs over 3,229 computer science papers. It proposes CATTS, which uses paper titles as an auxiliary training signal, and reports improvements over strong extractive and abstractive baselines on automated and human evaluations.
Key findings
- SciTLDR contains 5,411 TLDRs over 3,229 papers: 1,992 training papers with one gold TLDR each, 619 dev papers with 1,452 TLDRs, and 618 test papers with 1,967 TLDRs. Its summaries average 21 words, with a compression ratio of 238.1 and 15.2% novel words.
- Author-written TLDRs average 18.9 words and peer-review-derived TLDRs average 22.9 words; lexical overlap is very low (1-, 2-, and 3-gram mean Jaccard indices of 15.0%, 2.5%, and 0.7%). TLDR-PR is more abstractive, with 20.2% novel words versus 9.6% for TLDR-Auth.
- CATTS improves over BART baselines: in abstract-only input, CATTS gains +0.5 Rouge-1 and CATTS_XSUM gains +1.8 Rouge-1; in AIC input, CATTS gains +2.0 Rouge-1 and CATTS_XSUM gains +0.9 Rouge-1. In AIC, CATTS reaches 44.9 Rouge-1, 22.6 Rouge-2, and 37.3 Rouge-L versus BART’s 42.9/20.8/35.1.
- Human evaluation found CATTS_XSUM more informative than BART_XSUM (MRR 0.54 vs 0.42), comparable to gold TLDR-Auth (0.53) but below TLDR-PR (0.60). Correctness did not differ: both models averaged 2.5, with 42 ties, 10 cases favoring BART_XSUM, and 12 favoring CATTS_XSUM.
Methods
- Dataset construction combined author-written TLDRs collected from OpenReview with expert-derived TLDR-PR summaries created by rewriting the first 128 words of peer review comments. Twenty-eight University of Washington undergraduate computer science students were recruited, 20 continued after initial training, and annotators were paid $20/hour; full texts were parsed with S2ORC.
- Baselines included extractive PACSUM, BERTSUMEXT, and MatchSum, plus abstractive BART-large and BART-large finetuned on XSUM. CATTS appends control codes for TLDR and title generation, trains with an additional 20K arXiv paper-title pairs, and upsamples TLDR instances; experiments use abstract-only and abstract/introduction/conclusion (AIC) input spaces.
- Evaluation uses max Rouge over multiple gold TLDRs per paper, plus human evaluation: nugget-based informativeness on 76 sampled gold papers and author-rated correctness from 29 unique authors covering 64 arXiv papers.
Limitations
- Stated in the paper: experiments are limited to abstract-only and AIC input spaces, though full texts are provided for future work; the idea of a TLDR may differ across academic disciplines and is left for future work; a small fraction (<5%) of papers lacked available PDFs and were included using abstract-only text.
- Evident from the text: the dataset is restricted to computer science papers, and the human correctness evaluation is small (29 authors, 64 papers) and found no difference between BART_XSUM and CATTS_XSUM.
- Evident from the text: informativeness analysis uses only 76 sampled gold papers, and automated evaluation relies on max Rouge over multiple gold TLDRs, which rewards matching any gold summary and may not fully capture factual correctness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
35 Own result · 15 No basis given · 9 Own method · 9 Own interpretation · 5 Partly supported · 4 Sources not found · 2 Supported · 2 No full text of the sources · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As evidenced by earlier work in summarization evaluation (Cohan and Goharian, 2016), variability in human-written summaries (Zechner, 1996; Harman and Over, 2004) can negatively impact the reliability of automated summarization metrics like Rouge (Lin, 2004).”
What the sources say
- Not matchedcites Cohan and Goharian, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Zechner, 1996 source
Section 4.3 reports mean interhuman precision and recall of 0.49 when each subject’s selected sentences were compared with the others, showing variability among human-produced extracts. The citation to Zechner is attached to that variability; the sentence cites other works for the evaluation and Rouge points. Page numbers are not available in the supplied copy. (LLM, conf=0.87)
- Not matchedcites Lin, 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“We demonstrate that control codes can be used to effectively incorporate scaffold tasks (e.g. title generation) for denoising autoencoders like BART (Lewis et al., 2020).”
What the sources say
- Supportedcites Lewis et al., 2020 source
Section 2 (Model; page not available) identifies BART as a denoising autoencoder, which is what the citation establishes in the citing sentence. The control-code result is presented as the citing authors’ own demonstration, not as a finding attributed to Lewis et al. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is unlike the majority of existing summarization datasets that assume only one gold summary for a given document.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also observe through qualitative examples in Figure 3 how TLDR-Auth and TLDR-PR can differ greatly, even when they contain the same information content.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, TLDRPR focus more on articulating problems the paper addresses ( P ).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The training set contains 1,992 papers, each with a single gold TLDR.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interestingly, TLDR-PR place less emphasis on R and V in favor of further methodological details in the paper D .”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers.”
- Also stated in: § 1 Introduction · p. 1 · plain
“we introduce SCITLDR, a new dataset of 5,411 TLDRs of computer science papers.”
- Also stated in: § 2 Dataset construction · p. 2 · plain
“We introduce SCITLDR, a new multitarget dataset of 5,411 TLDRs over 3,229 scientific papers in the computer science domain.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, they are the most frequently co-occurring nuggets, appearing in 63% of TLDRAuth and 71% of TLDR-PR.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We introduce CATTS (Controlled Abstraction for TLDRs with Title Scaffolding), a simple yet effective method for learning to generate TLDRs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Titles often contain key information about a paper.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 28
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The goal of existing work in summarization of scientific documents is to generate abstracts or provide complimentary summaries to abstracts. (Collins et al., 2017; Cohan et al., 2018; Chandrasekaran et al., 2019; Yasunaga et al., 2019).”
What the sources say
- Not matchedcites Collins et al., 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Cohan et al., 2018 source
The Introduction describes this paper’s model as generating summaries of scientific papers. The Data section says the papers’ abstracts serve as ground-truth summaries, supporting the claim’s abstract-generation alternative. (LLM, conf=0.89)
- Partly supportedcites Chandrasekaran et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes citation-based mini-summaries, and the Task section (p. 2) asks systems to generate a structured summary of a scientific paper. It lists abstracts alongside community and human-written summaries, but does not say that the task aims to generate abstracts or that its summaries are intended to complement abstracts. It also does not characterize the goals of existing work generally. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification (Swayamdipta et al., 2018b; Cohan et al., 2019).”
What the sources say
- Not matchedcites Swayamdipta et al., 2018b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Cohan et al., 2019 source
The Introduction and §2.1 describe auxiliary structural scaffold tasks in a multitask framework for citation-intent classification. The Results (§4.3) report improved classification performance when the scaffolds are added. This supports the text-classification part of the sentence for which Cohan et al. is cited. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“We use BART-large, which achieves state-of-the-art results in summarization on XSUM.”
What the sources say
- Supportedcites Lewis et al., 2020 source
Section 5.1 identifies the model used in the large-scale experiments as the large BART model. Section 5.3 reports that BART outperformed the best previous XSum summarization system by roughly 3.5 points on all ROUGE metrics, supporting the state-of-the-art claim. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to multitask learning, training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models (Keskar et al., 2019; ElSahar et al., 2020; Sudhakar et al., 2019; Li et al., 2020).”
What the sources say
- Supportedcites Keskar et al., 2019 source
The Introduction (p. 1) describes training CTRL on varied text sources with control codes derived from their associated structure. Language Modeling with CTRL (p. 2) identifies it as an autoregressive conditional language model, and Control Codes (p. 5) shows that the codes enable predictable variation and increasingly constrain generation. (LLM, conf=0.89)
- Partly supportedcites ElSahar et al., 2020 sourceExperimental — accuracy not yet measured
Section 4 (p. 4) says the authors enrich reviews with category-derived control codes during training. Section 7 (pp. 8–9) reports that removing the codes degrades summary quality and that generated summaries tend to follow appropriate codes. The paper supports improved control in its review-summarization model, but does not isolate heterogeneous training data as the reason for that improvement or establish the result for autoregressive language models generally. (LLM, conf=0.84)
- Partly supportedcites Sudhakar et al., 2019 sourceExperimental — accuracy not yet measured
Section 2.3.2 describes an autoregressive transformer whose B-GST input includes a special token for the target style, and Section 4.3 reports improved style-transfer results. But the paper does not show that training on heterogeneous data annotated with control codes caused that improvement; Section 2.3.3 describes reconstruction training for style transfer, not a comparison of heterogeneous control-code training against an alternative. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While our multi-target setting allows us to mitigate some of the limitations of Rouge (Conroy et al., 2011; Cohan and Goharian, 2016), we acknowledge that relying only on automated metrics is insufficient for evaluating the quality of the models.”
What the sources say
- Supportedcites Conroy et al., 2011 source
The citation is attached to the limitations of ROUGE, which the paper documents directly. In Section 3, the authors say the gap between automatic and manual evaluation persists and can force the use of manual evaluation; the Conclusion identifies linguistic quality as a deficiency of ROUGE. These passages also support the citing sentence’s broader caution about relying only on automated metrics for summary evaluation. (LLM, conf=0.91)
- Not matchedcites Cohan and Goharian, 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that SCITLDR is smaller in comparison to automatically collected datasets, such as XSUM and ArXiv, but is larger in comparison to other manually collected datasets, such as SciSummNet.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this, SCITLDR contains TLDRs written from the perspective of the author ('TLDRAuth') and TLDRs written from the perspective of the peer reviewer('TLDR-PR').”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden.”
- Also stated in: § 1 Introduction · p. 2 · plain
“where the latter are collected using a novel annotation protocol that produces highquality summaries while avoiding the burden of reading the full paper.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: Scaling up data collection in a specialized scientific domain is costly and challenging.
- The authors state this as an assumption: The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).
- Also stated in: § 1 Introduction · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite similarities in length, the 1-, 2-, and 3-gram mean Jaccard indices between TLDR-Auth and TLDR-PR are 15.0%, 2.5%, and 0.7%, respectively, indicating extremely little lexical overlap between the two sources of TLDRs.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dev and test sets contain 619 and 618 papers each, with 1,452 and 1,967 TLDRs, respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most TLDRs contain between two to four nuggets (never all six), and will provide some indication of their subject area ( A ) and the paper's contributions ( C ).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that SCITLDR is more abstractive compared with other scientific domain datasets but less abstractive compared with non-scientific domain datasets.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe that SCITLDR has short summaries, like XSUM and NewsRoom, with long source documents, like BigPatent and the other scientific-domain datasets. This results in a much higher compression ratio compared with existing datasets.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TLDR-Auth tend to include results or scientific/theoretical findings ( R ) and often signal the value of their work ( V ) by describing their contributions as novel or their results as strong or state-of-the-art .”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we note that TLDR-Auth are on average 18.9 words long, while TLDR-PR are slightly longer on average at 22.9 words.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CATTS improves upon strong baselines under both automated metrics and human evaluations.”
- Also stated in: § 1 Introduction · p. 2 · boosted
“We show that CATTS applied to BART (Lewis et al., 2020), a state-of-the-art summarization model, results in performance improvement in both automated metrics and human evaluation.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 2 · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“TLDR-PR is more abstractive with a novelty score of 20.2% compared with TLDRAuth with a novelty score of 9.6%, where novelty is computed as the percentage of words in the TLDR not in the source paper.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While this up-sampling may indicate that CATTS is training on more TLDRs than BART, we allow BART training up to 20 epochs and it quickly overfits within a few epochs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average combined length of these contexts is 993 words and resulting compression ratio is 47.3, which is still higher than other datasets surveyed in Table 1.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that MatchSum has the highest extractive performance, followed by BERTSUMEXT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Curiously, despite both achieving the same Rouge-1, the generated TLDRs are quite different.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe that increasing input space from abstract-only to AIC greatly improves PACSUM 15 performance but decreases performance of both BERTSUMEXT and MatchSum.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The 'two hidden layer' detail by BART XSUM is from the paper introduction and the 'defining the appropriate sampling distributions' from CATTS XSUM is from the conclusion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For a given abstractive baseline BART or BART XSUM , our CATTS learning strategy results in improvements in both abstract-only and AIC settings.”
- Also stated in: § 6.1 Quantitative results · p. 8 · plain
“Comparing CATTS variants with their corresponding BART baselines, we observe that in the abstract-only setting, CATTS and CATTS XSUM achieve +0.5 and +1.8 Rouge-1, respectively. In the AIC setting, CATTS and CATTS XSUM achieve +2.0 and +0.9 Rouge-1, respectively.”
- Also stated in: § 8 Conclusion · p. 9 · plain
“We show that our approach improves over strong modeling baselines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.1 Quantitative results · p. 8 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our full text oracle achieves 54.5 Rouge-1, 30.6 Rouge-2, and 45.0 Rouge-L on the test set.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The average length of an abstract is 159 words and resulting compression ratio is 7.6.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Table 5, we see the systems all generate TLDRs of similar length to the average length reported in Table 1.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“PACSUM using the full text yields a Rouge-1 of 12.7, significantly worse than abstract-only.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BART XSUM and CATTS XSUM both achieve Rouge-1 of 40.7 on this paper.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Summarization in higher compression settings is challenging as it requires capturing more precisely the salient aspects of the document (Grusky et al.).”
What the sources say
- Not matchedcites Grusky et al.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Writing a TLDR of a scientific paper requires expert background knowledge and understanding of complex domain-specific language to identify the salient aspects of the paper, while maintaining faithfulness to the source and correctness of the written summary.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 1 · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually assessed every written summary, discarding TLDRs that did not adhere to the guidelines, and allowed 20/28 students who performed well to continue work beyond the first hour.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach addresses two main challenges: (1) the limited size of the training data and (2) the need for domain knowledge in order to write high-quality gold TLDRs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is not unexpected because TLDR-PR are derived from peer review comments which themselves have already gone through one stage of abstraction.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“To address the issue of multiple hypothesis testing over Rouge scores, we perform a Holm-Bonferroni (Holm, 1979) 16 correction for determining significant p -values in Table 3.”
What the sources say
- Not matchedcites Holm, 1979
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“SCITLDR is a new form of extreme summarization focused on scientific papers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised models like BERTSUMEXT and BART are trained on SCITLDR and the best model checkpoint chosen using dev set loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this setting, we have both chosen the strongest baseline and controlled for XSUM pretraining.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.
- Rests on an assumption: The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use the two-sided paired t-test against a null hypothesis of no difference to assess these differences.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since TLDRs often contain information spread across multiple sentences, we expect abstractive summarization methods to produce strong results for this task.”
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the multiple target summaries in SCITLDR reflect diverse perspectives and can be used to support summarization research into training and evaluation techniques previously unavailable with existing datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Best hyperparameters for the models are selected based on dev set Rouge-1.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BART XSUM focuses on the methodological contribution while CATTS XSUM focuses on a scientific finding.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We received responses from 29 unique authors with annotations covering 64 arXiv papers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, while our experiments are limited to abstract-only and AIC input spaces, we provide the full text of the source papers to support research into using longer input contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Scaling up data collection in a specialized scientific domain is costly and challenging.
“Scaling up data collection in a specialized scientific domain is costly and challenging.”
The authors state this as an assumption.
The typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph).
“Assuming the typical peer reviewer has carefully scrutinized the source paper and provided a faithful summary in their comment (often in the first paragraph), domain experts can rewrite these comments into TLDRs.”
The paper relies on this without support.
Titles often contain key information about a paper.
“As titles often contain key information about a paper, we hypothesize that training a model to generate titles will allow it to learn how to locate salient information in the paper that will be also useful for generating TLDRs.”
The paper relies on this without support.
All papers have a title.
“In addition, all papers have a title, and thus we have an abundant supply of paper-title pairs for training.”
The paper relies on this without support.
Human experts in computer science can reliably assess informativeness and correctness of system-generated TLDRs.
“we also have human experts in computer science assess system-generated TLDRs under two criteria - informativeness and correctness.”
The paper relies on this without support.
Original authors of papers can reliably assess the correctness of system-generated TLDRs for their papers.
“we ask the original authors of papers to assess the correctness of our system-generated TLDRs.”
The paper relies on this without support.
The 51 sampled papers used in human evaluation are representative of the SCITLDR test set.
“We perform our human evaluation on BART XSUM and CATTS XSUM using the AIC input space on 51 sampled papers.”
The paper relies on this without support.
The 76 gold papers used for nugget analysis are representative.
“we perform the nuggetbased analysis for information content over systemgenerated TLDRs for the same 76 gold papers from Section 3.2.”
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 10×
“PACSUM is an extension of TextRank (Mihalcea and Tarau, 2004), utilizing directed edges and a pretrained BERT model as a sentence encoder.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“To address the issue of multiple hypothesis testing over Rouge scores, we perform a Holm-Bonferroni (Holm, 1979)16 correction for determining significant p-values in Table 3.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“The first two use BERT-base and the last one uses RoBERTa-base (Liu et al., 2019).”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 5×
“We particularly focus on BART-large, a state-of-the-art pretrained model with strong results in text generation and summarization (Lewis et al., 2019).”
BackgroundMethodInfluentialTextRank: Bringing Order into Text
Mihalcea, R. F., & Tarau, P. (2004) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 3,631
Research works printed but not linked: 3
- Paul Over. 2003. An introduction to duc 2003: Intrinsic evaluation of generic news text summarization systems. In Proceedings of Document Understanding Conference 2003 .
- In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 32693279, Hong Kong, China. Association for Computational Linguistics.
- Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018b. Syntactic scaffolds for semantic structures. In EMNLP .
Cited by
Sorted by citations
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,852
“Cachola et al. [3] were able to collect a dataset of 5.4K Too Long; Didn’t Read (TLDR) summaries of 3.2K machine learning papers.”
Bulathwela, S., Pérez-Ortiz, M., Holloway, C., Cukurova, M., & Shawe-Taylor, J. (2024) · Sustainability · doi:10.3390/su16020781 · Cited by 269
“The generative capabilities of linguistic content (e.g., automatic question generation [41,42], personalised feedback [43] and summarisation [44]) have seen significant progress.”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
“Recent efforts to expand the application of summarization to new domains such as legal (Kornilova and Eidelman, 2019) or scholarly (Cachola et al., 2020) text will hopefully lead to the study of inconsistency detection in these novel domains, and perhaps even out of summarization on tasks such as text simplification,…”
BackgroundCyberbullying detection for low-resource languages and dialects: Review of the state of the art
Mahmud, T., Ptaszynski, M. E., Eronen, J., & Masui, F. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103454 · Cited by 181
Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols
Head, A., Lo, K., Kang, D., Fok, R., Skjonsberg, S., Weld, D. S., & Hearst, M. A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445648 · Cited by 91
“angers [26], and crowds [39] to annotate and discuss passages of papers. Other approaches to saving the scientist time include tools to support literature search (e.g., [77, 80, 109]), summarize text [10, 87], or rewrite passages in simpler language [46]. 2.3.2 Links within Papers. Reading interfaces can also assist r…”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Cachola, I., Lo, K., Cohan, A., & Weld, D.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Published
- 2020
- Pages
- 4766-4777
- OpenAlex
- W3021892802
- Semantic Scholar
- 3502a542b2e98d9094e1880a30f652d4170b9534
- Cited by
- 169
- References
- 51
- Citations by year
- 2026: 6 · 2025: 18 · 2024: 26 · 2023: 48 · 2022: 35 · 2021: 34 · 2020: 2
- FWCI
- 11.35
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1936940, 2040196)
- University of Washington
- Office of Naval Research (N00014- 18-1-2193, N00014-18, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Linguistics and Language › Text Readability and Simplification
- Keywords
- annotation protocol, computer science literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
