The DOI matches its Crossref record; 60 of the 63 references it cites were found in a registry or an index; 28 of the 31 sources its claims cite were checked for support.
RARR: Researching and Revising What Language Models Say, Using Language Models
Open accessAbstract
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
RARR is a model-agnostic research-and-revise system that retrieves web evidence for an LM-generated passage and post-edits unsupported content while preserving the original text. Across factoid QA, reasoning chains, and dialog, RARR improves attribution more robustly than prior edit models and preserves original intent over 90% of the time.
Key findings
- On PaLM NQ outputs, RARR raises auto-AIS from 45.6 to 54.9 and human AIS from 35.4 to 43.4, with intent preservation 90.0, Levenshtein preservation 89.6, combined preservation 83.1, and F1_AP 57.0.
- On PaLM SQA outputs, RARR raises auto-AIS from 37.6 to 45.1 and AIS from 24.5 to 31.5, with intent preservation 92.6, Levenshtein 89.9, combined 84.6, and F1_AP 45.9.
- On LaMDA QReCC outputs, RARR raises auto-AIS from 18.8 to 29.4 and AIS from 13.2 to 28.3, with intent preservation 95.6, Levenshtein 80.2, combined 78.1, and F1_AP 41.5.
- RARR is the only method reported to perform robustly across all three datasets; EFEC and LaMDA preserve original intent only 6–40% of the time, while RARR preserves intent over 90% of the time. Editing increases attribution by up to 13% absolute while changing only 10–20% of the text.
Methods
- The paper formalizes Editing for Attribution: given a generated passage x, output a revised passage y plus an attribution report A of evidence snippets. It evaluates attribution with sentence-level AIS and an automated NLI-based auto-AIS, and preservation with human intent preservation, Levenshtein similarity, and their combination.
- RARR uses few-shot prompting with PaLM 540B: comprehensive question generation, Google Search retrieval (K=5 pages, top J=1 evidence per query), an agreement model to detect disagreement, an edit model to revise only when needed, and selection of at most M=5 evidence snippets for the attribution report.
- Experiments use 150 development and 150 test passages for each generation-model/source-dataset combination: PaLM and GPT-3 on NQ and SQA, and LaMDA and GPT-3 on QReCC. Baselines include EFEC and LaMDA Research; human evaluation and automated metrics are both reported.
Limitations
- Stated: attribution and preservation may not deserve equal weight depending on the application; the metrics do not capture all aspects of attribution, such as self-evident sentences, varying assertion scope, directness, or implicatures; preservation metrics do not cover discourse or logical coherence, and heavily flawed inputs may require changes that the metrics penalize.
- Stated: RARR is not fully general—it is not prepared for poetry or long documents without prompt changes—and it tends to preserve rather than delete claims it cannot attribute, even when those are hallucinations. It is also computationally costly because it prompts a large language model.
- Evident from the text: the main experiments are based on a single run, though Appendix A reports variance over 3 random runs; the paper also notes that evidence trustworthiness and conflicting evidence are not addressed by the current implementation or evaluation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
49 Own result · 18 Own interpretation · 18 No basis given · 12 Supported · 10 Own method · 8 Partly supported · 1 Sources not found · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We utilize the natural language inference (NLI) model from Honovich et al. (2022), which correlates well with AIS scores.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
What the sources say
- Partly supportedcites Honovich et al. (2022) sourceExperimental — accuracy not yet measured
Section 3.3 and Appendix B describe an NLI model trained by Honovich et al. (2022), supporting the model identification. Section 4 reports its performance using ROC AUC on factual-consistency datasets, but the copy does not report a correlation with AIS scores (searched: AIS scores, Attributable to Identified Sources). (LLM, conf=0.86)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite these incredible advances, state-of-theart LMs still frequently produce biased, misleading, or unsupported content, colloquially called”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Partly supportedcites Maynez et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.1 defines hallucinations as summary content unsupported by or misrepresenting the source document, and the assessment in Section 5.2 finds substantial hallucinations in the summarization systems studied. The paper does not establish that language models generally frequently produce biased content; its findings concern abstractive summarization. (LLM, conf=0.87)
- Partly supportedcites Menick et al., 2022 sourceExperimental — accuracy not yet measured
The opening passage (p. 1) says large language models can hallucinate convincing nonsense, and the Conclusion (p. 18) says they often produce hallucinated facts. The Limitations section (p. 18) discusses misleading, cherry-picked quotations. The paper does not establish that language models frequently produce biased content; its discussion of bias concerns supporting sources such as Wikipedia. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Even retrievalaugmented models (Guu et al., 2020;Lewis et al., 2020), which retrieve relevant documents and then condition on them to generate text, still do not guarantee attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Supportedcites Guu et al., 2020 source
Appendix C (p. 11) reports that REALM can retain facts in its parameters and make predictions that do not update when the retrieved corpus changes. This supports the claim that retrieval alone does not guarantee a prediction is attributable to the retrieved documents. (LLM, conf=0.88)
- Supportedcites Lewis et al., 2020 source
The Introduction (p. 2) describes RAG as retrieving documents and conditioning generation on them. But H Retrieval Collapse (p. 19) reports cases where the generator learned to ignore those documents, supporting the claim that retrieval does not guarantee attribution to them. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In real-world scenarios such as the one which RARR operates in, relevant evidence may not be provided, necessitating retrieval (Fan et al., 2020;Piktus et al., 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Fan et al., 2020 source
Section 3.2 (Question Answering; page not specified) says crowdworkers receive a claim, its source, and a question, then search the web for information to answer it. Section 4 (QABrief Model; page not specified) likewise retrieves an evidence document for each question. These passages support the claim that relevant evidence must be retrieved rather than being provided at the outset. (LLM, conf=0.90)
- Supportedcites Piktus et al., 2021 source
Section 2.1 (p. 3) says knowledge-intensive tasks require access to an external knowledge source and are typically solved by retrieving relevant documents before generating an answer. This supports the cited point that relevant evidence may need to be retrieved rather than supplied with the task. (LLM, conf=0.90)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even if a revision preserves intent, it may still make superfluous modifications, such as reordering words, changing textual style, or including unnecessary additional information (Thorne and Vlachos, 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.2 Measuring preservation
What the sources say
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
Section R2—Supported by Evidence says corrections should target errors in the original claim rather than introduce unrelated information; Section 7 Evaluation also checks whether unrelated information was added. These passages support the concern about unnecessary additions, but the copy does not discuss intent-preserving revisions that needlessly reorder words or change textual style. No page numbers are available in the indexed passages. (LLM, conf=0.83)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Evidence-based Factual Error Correction (EFEC; Thorne and Vlachos, 2021) also implements a full research-and-revise workflow trained on Wikipedia passages (Thorne et al., 2018).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Thorne and Vlachos, 2021 source
The Retrieved Evidence section says EFEC retrieves evidence from Wikipedia and rewrites claims to better match it. Section 6 (Data) identifies the training basis as FEVER claims paired with Wikipedia evidence, supporting the description of a research-and-revise workflow trained on Wikipedia passages. Page numbers are unavailable in our copy. (LLM, conf=0.90)
- Partly supportedcites Thorne et al., 2018 sourceExperimental — accuracy not yet measured
The Introduction and §3.1 (page numbers unavailable) establish that FEVER’s claims and verification evidence come from Wikipedia. The cited paper does not address EFEC or establish that EFEC was trained on those passages. The research-and-revise workflow is attributed to the separate 2021 citation, not this one. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In this paper, we generally avoid the term”
- Also stated in: § E Details on the dataset · plain
“Even if a claim is attributed to a particular source, it does not guarantee that the source is”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
What the sources say
- Supportedcites Menick et al., 2022 source
Section 3.7 (pp. 16–17) distinguishes a claim being supported by evidence from its being true. It reports examples of false claims rated as supported and concludes that a high support score can coexist with a low truthfulness score. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § E Details on the dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we find that existing revision models do not always generalize across many tasks (and were not originally intended to), and therefore propose a new research-and-revise model that leverages the power of few-shot prompting in large language models to robustly generalize across domains.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By manually inspecting 30 examples from our benchmarks, we found M = 5 snippets to be sufficient for full attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only RARR has a robust F1 AP across all tasks.”
- Also stated in: § E Details on the dataset · plain
“Only RARR has a robust F1 AP across all tasks.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § E Details on the dataset · plain
- It may state two separate claims as one. 31
- Probably not a substantial claim. 36
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prior work has shown that retrieval-augmented models generate text that either includes additional information outside the retrieved documents (Dziri et al., 2022), ignores the documents altogether (Krishna et al., 2021), or even contradicts the documents (Longpre et al., 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Supportedcites Dziri et al., 2022 source
The Dziri citation is attached only to the claim that generated text includes information outside the retrieved documents, not to the later claims about ignoring or contradicting them. The Introduction reports that model-generated responses contain hallucinations, including unsupported objective factual information; Section 3 reports that the models generate and amplify hallucinations relative to their source knowledge. The paper describes these as knowledge-grounded conversational models using knowledge snippets rather than using the citing sentence’s term “retrieval-augmented.” (LLM, conf=0.90)
- Supportedcites Krishna et al., 2021 source
The Introduction states that the authors’ retrieval-augmented system does not actually use the documents it retrieves. Section 3.1 supports this with a control experiment: replacing predicted retrievals with random documents had almost no effect on generated answers. The Krishna et al. citation is attached specifically to the claim that models ignore retrieved documents, not to the sentence’s other two claims. (LLM, conf=0.94)
- Supportedcites Longpre et al., 2021 source
The Results passage under §3.3 Metrics reports that, when the provided passage contained a substituted answer, generative QA models sometimes predicted the original answer learned during training instead, contradicting the provided passage. The Longpre et al. citation is attached only to the claim about contradicting documents, not the other two behaviors in the sentence. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“LaMDA (Thoppilan et al., 2022) generates responses in three steps: 1) generate a “base response”; 2) generate search queries from the base response; 3) generate a “revised response” conditioned on the base response and retrieved evidence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Thoppilan et al., 2022 source
The fine-tuning description (p. 8) says LaMDA takes a response from the base model, generates a toolset query, and predicts a grounded response from the base response, query, and retrieved snippet. The example (p. 9) calls these the Base and Research phases; the Research phase can issue multiple queries before responding to the user. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“EFEC uses a neural retrieval model (Karpukhin et al., 2020) to retrieve from Wikipedia”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Karpukhin et al., 2020 source
The cited paper describes a neural dense-passage retriever: §3.1 says it uses BERT encoders to retrieve passages, and §4.1 identifies those passages as coming from Wikipedia. The citation identifies the retrieval model used by EFEC, rather than asking this source to document EFEC itself. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FRUIT (Logan IV et al., 2022) and PEER (Schick et al., 2022) both implement an editor that is fine-tuned on Wikipedia edit history with the goal of updating outdated information and collaborative writing respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Not matchedcites Logan IV et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Schick et al., 2022 source
The Schick et al. citation attaches to the PEER half of the sentence, not the FRUIT half. The Introduction (p. 2) describes PEER’s goal as collaborative editing, while Training Data (p. 5) and Experimental Setup (p. 6) say it was trained from Wikipedia edit history starting from a pretrained model. The paper identifies FRUIT separately as a dataset for updating Wikipedia texts (Downstream Tasks, p. 8). (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Language models can generate reasoning chains to answer complex questions (Wei et al., 2022).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.1 Evaluation setups
What the sources say
- Supportedcites Wei et al., 2022 source
Section 2 (p. 2) states that sufficiently large language models can generate chains of thought when given few-shot demonstrations, and gives a multi-step math question as an example. The claim says models can do this, not that all models do so without prompting. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A similar strategy has been employed to train textplanning models (Narayan et al., 2022).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Query generation
What the sources say
- Supportedcites Narayan et al., 2022 source
The Introduction says the authors use question-generation technology to create question–answer blueprints that serve as text plans. The Iterative Model section describes training a model to produce those blueprints while planning and generating text one sentence at a time. This supports the citing sentence’s characterization of a similar strategy used to train text-planning models. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previously, Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
What the sources say
- Supportedcites Rashkin et al. (2021) source
The Introduction (p. 2) explicitly introduces AIS as the authors’ evaluation framework. Definition 8 (p. 10) defines whether a sentence is AIS using conditions that either hold or do not, supporting the description of a binary notion of attribution; the paper also reports human evaluation studies in the Conclusion (p. 26). (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The approach is inspired by works on fact-checking 2 where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans (Thorne et al., 2018;Schuster et al., 2021;Thorne and Vlachos, 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Partly supportedcites Thorne et al., 2018 sourceExperimental — accuracy not yet measured
Section 3.2 (Task 2—Claim Labeling; page not supplied) describes annotators finding Wikipedia evidence to support or refute human-generated claims. This supports citing FEVER as fact-checking work, but the paper does not establish that a research-and-revise workflow is effective at attributing or correcting unattributed human claims. (LLM, conf=0.82)
- Partly supportedcites Schuster et al., 2021 sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says Wikipedia users verify and correct others’ revisions, and §3.1 describes collecting factual edits. The paper does not establish that simple research-and-revise workflows are effective, or address attributing previously unattributed human claims. (LLM, conf=0.82)
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
The Retrieved Evidence section reports that the system retrieves evidence and uses it to correct claims, with satisfactory corrections for some evaluated instances. The Conclusions and Future Work section says the evaluation used mutations introduced by human annotators and may not represent errors in real-world documents. The paper does not demonstrate attribution of unattributed claims, so that part and the broader implication about human-made claims are not supported; page numbers are unavailable in our copy. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“EFEC fine-tunes a T5-based model to revise text conditioned on multiple evidence snippets using both semi-supervised and fully-supervised approaches.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
Section 5.3 says the paper fine-tunes a T5 model to generate corrections from masked claims and evidence, and Section 4.1 specifies multiple retrieved evidence sentences. The paper calls its main training approach “distant supervision,” not semi-supervision; it also fine-tunes a separate fully supervised T5 model as a performance ceiling (Implementation, “Fully Supervised Ceiling”). (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We compare against their fully-supervised approach, which performed best in their experiments.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Thorne and Vlachos, 2021 source
The Results section states that the fully supervised models had the highest rate of satisfactory corrections that improved factuality. It also reports 68.9% satisfactory corrections with gold evidence, supporting the citing sentence’s comparison. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This form of question-guided agreement was previously explored by Honovich et al. (2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.2 Revision stage
What the sources say
- Supportedcites Honovich et al., 2021 source
Figure 2 describes generating questions from a response, answering them using grounding knowledge, and comparing the answers. The “Answer Similarity and Final Scores” section explains how those comparisons assess agreement between the response and the knowledge, supporting the paraphrase. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“An exception is LaMDA (Thoppilan et al., 2022), a language model for dialog that performs revision by training on human annotations.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Thoppilan et al., 2022 source
The Abstract (p. 1) identifies LaMDA as a language model specialized for dialog. Section 5 (p. 6) says crowdworkers edit model responses, and Section 6.2 (p. 8) says their corrections provide fine-tuning data that LaMDA learns to imitate. This supports the paraphrase that it learns to revise responses from human annotations. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FRUIT (Logan IV et al., 2022) and PEER (Schick et al., 2022) both implement an editor that is fine-tuned on Wikipedia edit history with the goal of updating outdated information and collaborative writing respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Not matchedcites Logan IV et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Schick et al., 2022 source
The Introduction (p. 2) describes PEER’s goal as a collaborative writing model. Training Data (p. 5) says its main training data comes from Wikipedia’s edit history, and Experimental Setup (p. 6) says PEER is initialized from LM-Adapted T5 and trained on approximately five million Wikipedia edits. Section 4 (p. 4) identifies PEER-Edit as the main model for editing. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results on GPT-3 passages show similar trends (Appendix A).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this case, RARR using CQGen needs to sacrifice more preservation score to edit the text, leading to a lower F1 AP score.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6.2 Ablations
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of F1 AP , RARR is the only method that performs robustly across all three datasets, and significantly outperforms prior methods on NQ and SQA.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attribution performance of sentences-as-queries drops significantly, while CQGen is more robust.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6.2 Ablations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The answer tends to be context-dependent, featuring pronouns and implicit references.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Knowledge-intensive dialogs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After editing, RARR can increase attribution by up to 13% absolute, while changing only 10-20% of the text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, more revision leads to less preservation score and spurious changes to the text passage, as demonstrated in Figure 8.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Ablating agreement model
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that RARR not only preserves the short answer accuracy but actually improves it by roughly 5%.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact on downstream task performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RARR significantly improves attribution while preserving most of the original text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in datasets where the claim contains multiple sentences (NQ and SQA), such a behavior yields low preservation scores, and also results in outputs that are less informative.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“however, not all passages in our experiments are supported by Wikipedia articles.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“EFEC reduces the passage x into a single sentence.”
- Also stated in: § E Details on the dataset · plain
“EFEC reduces the passage x into a single sentence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § E Details on the dataset · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LaMDA changes the writing style.”
- Also stated in: § E Details on the dataset · plain
“LaMDA changes the writing style.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Also stated in: § E Details on the dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EFEC frequently attempts to summarize the entire passage into one sentence, or drops later sentences.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also see that editing is crucial to improve attribution: if we only retrieve evidence to support the original response x without editing, attribution ranges from the low 10s to mid 30s.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting passages are mostly coherent but often contain factual errors.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.1 Evaluation setups
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we do not provide a means for judging trustworthiness, the design of RARR does allow for the research stage to restrict its search over a userspecified corpus, based on what the user deems trustworthy.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, RARR tends to preserve rather than delete claims that it cannot attribute.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that RARR's performance is low on examples with mathematical reasoning, as these are beyond the capability of the edit model with our current prompt.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sometimes the whole output is based on a false premise and needs to be completely rewritten, in which case RARR cannot satisfactorily edit due to our revision threshold (Section 3.2).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the AIS scores from human correlate well with auto-AIS scores, with some bias for non-attributed sentences to be judged as attributed by auto-AIS.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Comparing human and automated evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When RARR is not 100% successful in making text consistent with retrieved evidence, the revised text will be partially attributed.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, our model is computationally costly, since it is based on prompting a large language model.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The editor occasionally produces large edits that bring the new revision close to e but far from the current y.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EFEC and LaMDA also face this issue, but instead trade preservation for attribution and rewrite the text to a different topic.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“proves attribution of the explanations on all four categories of MMLU, although the increases are relatively small.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conducted correlation studies between human and automatic metrics and found strong Pearson correlation (attribution = 0.74; preservation = 0.62).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Comparing human and automated evaluation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This work does not address the question of whether a source is trustworthy, or the related topic of misinformation.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The standard deviations of Attr auto , Pres Lev , and F1 AP are 1.2, 0.5, and 1.0 respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > A Additional experiments and analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In fact, occasionally ignoring the retrievals can make the models more robust to bad retrievals (Khandelwal et al., 2020), illustrating that end-task performance and attribution are not always aligned.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Not matchedcites Khandelwal et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The main experiments in Section 5 are based on a single run.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > A Additional experiments and analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This helps reduce the editor's deviation from the current y.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.2 Revision stage
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This behavior generally increases the attribution score, because it is usually easier to make one sentence fully attributable than many sentences.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- Rests on an assumption: Most existing LMs lack a built-in mechanism for attribution.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We suspect that this may be due to noisy retrievals, which sometimes provide misleading evidence (exemplified in Figure 7d).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact on downstream task performance
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To more fairly compare the editing capabilities of EFEC, we instead use the evidence retrieved by our research stages (CQGen and web search).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The authors have manually looked through most of the data and found no personal identifiers.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In the main paper, we present two automatic metrics to conveniently gauge these properties, but rely on a human evaluation as the gold standard.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > C Details on human evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We reject an editing if the edit distance is more than 50 characters or more than half of the original text length.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We had 3 annotators rate each example in the pilot phase to measure inter-annotator agreement, and had a single rater annotate each example afterwards.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Rater recruitment and training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We tune our hyperparameters on the validation set as well.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These news article summaries are often about less widely known people and events, which is challenging for retrievers, leading to low attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that using web search with standard question generation methods may fail to capture important context from the input, and is not sufficient for the attribution task.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually wrote our prompts by eye-balling quality on a dozen of examples from a separate validation set.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Most existing LMs lack a built-in mechanism for attribution.
“Most existing LMs, such as those based on sequence-to-sequence architectures, lack a builtin mechanism for attribution.”
The paper relies on this without support.
Pres intent requires human annotation.
“Since Pres intent requires human annotation, we use Pres Lev as an automated metric for model development.”
The paper relies on this without support.
Gold or reference edits are often only available for specialized domains.
“which are often only available for specialized domains.”
The paper relies on this without support.
After retrieving evidence, some parts of the passage may be properly attributed, but other parts remain unattributed and should be revised.
“After retrieving evidence, certain parts of x may now be properly attributed, but other parts remain unattributed and should be revised.”
The paper relies on this without support.
A prompt with six human demonstrations is sufficient for PaLM to adequately learn the comprehensive question generation task.
“A prompt with six human demonstrations was sufficient for PaLM to adequately learn the task.”
The paper relies on this without support.
LaMDA was not optimized for preservation.
“since LaMDA was not optimized for preservation.”
The paper relies on this without support.
EFEC's training data was limited to single sentences.
“which was limited to single sentences.”
The paper relies on this without support.
The 40 manually inspected examples are representative of the question generation model's coverage.
“a manual inspection of 40 examples shows > 80% with questions that fully cover all aspects of the input text.”
The authors state this as an assumption.
Manually editing examples until judged 100% attributable approximates the minimal amount of editing needed to make a text passage fully attributed.
“As an approximation, we manually edited 30 examples in our NQ benchmark until we judged them to be 100% attributable.”
The paper relies on this without support.
Attribution and preservation are both subjective properties that may change with even small edits.
“Attribution and preservation are both subjective properties that may change with even small edits.”
The paper relies on this without support.
There is currently no linguistic consensus on what constitutes a claim.
“While each sentence may contain multiple claims that could be attributed independently, there is currently no linguistic consensus on what constitutes a claim.”
The paper relies on this without support.
Utterances can make assertions with varying levels of directness.
“In addition to varying scope, utterances can also make assertions with varying levels of directness.”
The paper relies on this without support.
Some content may be simultaneously supported by certain sources, while contradicted by others.
“There is also the possibility that some content may be simultaneously supported by certain sources, while contradicted by others.”
The paper relies on this without support.
Few-shot prompting has proven to be effective for many recent large language models.
“Few-shot prompting has proven to be effective for many recent large language models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, we can do this without a separate claim decomposition step.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Gao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Gao et al. (2023); Min et al. (2023); Wang et al. (2023) decompose each generated sentence into atomic facts and then search for relevant documents to support each atomic fact.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Gao et al. (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent studies (Gao et al. 2023; Chern et al. 2023) highlight the potential of LLMs to detect factual content in texts.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Gao et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, methods that use search engines only for self-correction (Gao et al., 2023; Yu et al., 2023) are unfair because they can use search engines to directly improve the initial response generation.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“A popular source of knowledge is search engines, which are often used with queries generated from initial responses to retrieve information for validating their correctness (Gao et al., 2023;Jiang et al., 2023b).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“As shown in Table 5, some prior studies (Gao et al., 2023;Yu et al., 2023;Zhao et al., 2023) use external knowledge only for selfcorrection, while they can also directly use external knowledge to improve the initial response generation process.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
In the citing paper
“For example, RARR (Gao et al., 2023) uses external knowledge to detect mistakes in initial responses, while it does not use any external knowledge when generating initial responses.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
“After editing, RARR can increase attribution by up to 13% absolute, while changing only 10-20% of the text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model auto-AIS AIS intent Lev comb F1 AP
“RARR 18.8 → 29.4 13.2 → 28.3 95.6 80.2 78.1 41.5”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
“We found that RARR is the only method that preserves the original intent of x over 90% of the time”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model auto-AIS AIS intent Lev comb F1 AP
“RARR 45.6 → 54.9 35.4 → 43.4 90.0 89.6 83.1 57.0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact of model scale
“We also experimented with keeping the editor stage at 540B while shrinking the query generation stage to 64B”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
“Full RARR qgen 62B, editor 540B”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
“One could identify unattributed parts using either the automated attribution score (Attr AIS ) or the relevance scores used to generate the attribution report (Section 3.3).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
“During model development, we define an automated metric, auto-AIS (Attr auto ), that approximates human AIS judgments.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Research works printed but not linked: 1
- Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In NAACL. · DOI
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“To enhance the trustworthiness and attribution of LLMs, Gao et al. (2023a) adopt the research-then-revise workflow, which initially research relevant evidence and subsequently revise the initial generation based on detected discrepancies with the evidence.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
“…to enhance information accuracy [143, 238], increasing model transparency to foster understanding of decision processes [86, 202, 203], developing fact-checking systems to verify outputs [200, 201], and designing hallucination detection tasks [325, 326] to evaluate and improve model performance.”
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“To enhance the trustworthiness and attribution of LLMs, Gao et al. (2023a) adopt the research-then-revise workflow, which initially research relevant evidence and subsequently revise the initial generation based on detected discrepancies with the evidence.”
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
“This section will provide a range of evaluation metrics and benchmarks for generative information retrieval methods, along with analysis and discussions on their performance.”
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting
Guan, X., Liu, Y., Lin, H., Lu, Y., He, B., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29770 · Cited by 125
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
Wang, Y., Pan, Y., Su, Z., Deng, Y., Zhao, Q., Du, L., Luan, T. H., Kang, J., & Niyato, D. T. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3576176 · Cited by 115
“For instance, RARR [206] utilizes search engines to gather relevant external knowledge and refine responses, while LURE [207] reduces hallucinations by correcting object-level inaccuracies during post-processing.”
GeneAgent: self-verification language agent for gene-set analysis using domain databases
Wang, Z., Jin, Q., Wei, C.-H., Tian, S., Lai, P., Zhu, Q., Day, C., Ross, C., Leaman, R., & Lu, Z. (2025) · Nature Methods · doi:10.1038/s41592-025-02748-6 · Cited by 114
“This cascading structure improves the traditional step-by-step chain-of-thought reasoning process 23 , enabling autonomous verification of the inference process 24 .”
Retrieving Supporting Evidence for Generative Question Answering
Huo, S., Arabzadeh, N., & Clarke, C. L. A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625336 · Cited by 75
“However, unlike RARR which makes uses of few-shot prompting [6] and external query-document relevance model, our experiments use nothing else besides the LLM itself and a retrieval pipeline.”
MethodResultDSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 911
“The formatting of these example prompts is adapted from Gao et al. (2023a).”
BackgroundIdentifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57
“A common approach here is to prompt the LLM back with feedback on how grounding fails, as shown in [28]:”
BackgroundBiomedRAG: A retrieval augmented large language model for biomedicine
Li, M., Halil Kilicoglu, B., Xu, H., & Zhang, R. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104769 · Cited by 67
Transparency of medical artificial intelligence systems
Kim, C., Gadgil, S. U., & Lee, S. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00363-w · Cited by 60
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 78
“Another IR approach is to fact-check a system’s outputs and remove errors (Gao et al., 2023; Jiang et al., 2023).”
BackgroundChain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42
“Other approaches include using tools for fact-checking (Chern et al., 2023a; Galitsky, 2023; Peng et al., 2023), or linking to external documents with attribution (Menick et al., 2022; Rashkin et al., 2023; Gao et al., 2023).”
BackgroundMethodHalluLens: LLM Hallucination Benchmark
Bang, Y., Ji, Z., Schelten, A., Hartshorn, A., Fowler, T., Zhang, C., Cancedda, N., & Fung, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1176 · Cited by 41
HaluCheck: Explainable and verifiable automation for detecting hallucinations in LLM responses
Heo, S., Son, S., & Park, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126712 · Cited by 55
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 82
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
Enhancing Accuracy in Large Language Models Through Dynamic Real-Time Information InjectionPreprint
Ouyang, Q., Wang, S., & Wang, B. (2023) · Preprints.org · doi:10.20944/preprints202312.1987.v1 · Cited by 36
A Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33
Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques
Son, M., Won, Y.-J., & Lee, S. (2025) · Applied Sciences · doi:10.3390/app15031430 · Cited by 35
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 112
Quantum Natural Language Processing
Widdows, D., Aboumrad, W., Kim, D., Ray, S., & Mei, J. (2024) · Künstliche Intell. · doi:10.1007/s13218-024-00861-w · Cited by 38
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 66
Mitigating reasoning hallucination through Multi-agent Collaborative Filtering
Shi, J., Zhao, J., Wu, X., Xu, R., Jiang, Y.-H., & He, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125723 · Cited by 34
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29
Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety
Gaur, M., & Sheth, A. (2024) · AI Magazine · doi:10.1002/aaai.12149 · Cited by 34
Exploring Data Analysis Methods in Generative Models: From Fine-Tuning to RAG Implementation
Guțu, B. M., & Popescu, N. (2024) · Computers · doi:10.3390/computers13120327 · Cited by 20
Reducing hallucinations of large language models via hierarchical semantic piece
Liu, Y., Yang, Q., Tang, J.-W., Guo, T., Chen, W., Pan, L., Sai, X., Gao, X., Li, Z., Liu, J., et al. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01833-9 · Cited by 21
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Knowledge-Augmented Language Model Verification
Baek, J., Jeong, S., Kang, M., Park, J. C., & Hwang, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.107 · Cited by 33
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143
Structured knowledge-based causal discovery: Agentic streams of thought
Meier, S., Raut, P. N., Mahr, F., Thielen, N., Franke, J. E., & Risch, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104202 · Cited by 14
Qlarify: Recursively Expandable Abstracts for Dynamic Information Retrieval over Scientific Papers
Fok, R., Chang, J. C., August, T., Zhang, A. X., & Weld, D. S. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676397 · Cited by 46
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 32
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
“Then, automated algorithms or human raters can examine whether the reference source really supports the generated answers using different criteria (Gao et al. 2023; Yue et al. 2023; Rashkin et al. 2023; Bohnet et al. 2023; Menick et al. 2022).”
TextGrad: Automatic "Differentiation" via Text
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07496 · Cited by 12
Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
Sun, X., Li, J., Zhong, Y., Zhao, D., & Yan, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889448 · Cited by 24
“Consequently, researchers have explored post-processing approaches [10]–[14] involving hallucination detection or correction post-content generation.”
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Ravichander, A., Ghela, S., Wadden, D., & Choi, Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.71 · Cited by 13
Fine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14
Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models
Sicari, S., Moreno, J. F. C., Rizzardi, A., & Coen‐Porisini, A. (2024) · ACM Computing Surveys · doi:10.1145/3703454 · Cited by 13
LLM Critics Help Catch LLM Bugs
McAleese, N., Pokorny, R. M., Uribe, J. F. C., Nitishinskaya, E., Trebacz, M., & Leike, J. (2024) · arXiv · doi:10.48550/arxiv.2407.00215 · Cited by 10
Social-RAG: Retrieving from Group Interactions to Socially Ground AI Generation
Wang, R., Zhou, X., Qiu, L., Chang, J. C., Bragg, J., & Zhang, A. X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713749 · Cited by 23
Question Answering as Programming for Solving Time-Sensitive Questions
Zhu, X., Yang, C., Chen, B., Li, S., Lou, J., & Yang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.787 · Cited by 11
AutoMix: Automatically Mixing Language Models
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.12963 · Cited by 19
“Finally, different from works that rely on external knowledge bases for verifying the outputs of language models (Peng et al., 2023; Gao et al., 2023; Pan et al., 2023b), AutoMix uses the context supplied with the question to verify the answer.”
MethodOn the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 40
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 13
“…to external corpora [Petroni et al., 2021] and utilizes a retriever component for knowledge augmentation [Ram et al., 2023, Shi et al., 2023, Rashkin et al., 2021, Gao et al., 2022, Bohnet et al., 2022, Menick et al., 2022] to improve their ability to provide comprehensive and accurate answers.”
BackgroundVariational Dehallucination via Retrieval-Augmented Prompt Learning
Lin, H.-Y., Chang, Y.-C., & Chien, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3646477 · Cited by 4
“In addition, the postgeneration approach [38] reversed this order by generating text first and then retrieving additional information to validate or refine the initial output.”
Effective Tool Augmented Multi-Agent Framework for Data Analysis
Zhang, X., Mao, Z., Chen, Z., & Gao, S. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0013 · Cited by 7
“Gao [40] proposed an approach named RARR, which utilizes retrieval to revise the correctness of input text and preserve the original input as much as possible through post-editing.”
MethodMeasuring the Influence of Incorrect Code on Test Generation
Huang, D., Zhang, J. M., Harman, M., Du, M., & Cui, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764532 · Cited by 21
Concise Answers to Complex Questions: Summarization of Long-form Answers
Potluri, A., Xu, F., & Choi, E. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.541 · Cited by 7
Gupta, D., Bartels, D., & Demner‐Fushman, D. (2025) · Scientific Data · doi:10.1038/s41597-025-05233-z · Cited by 9
“The approaches, Retrieval Augmented Generation (RAG) 5 and retrofit attribution 6 , provide sources to the models to guide answer generation or to find support and post-edit the generated output, respectively.”
Enhancing LLM’s Reliability by Iterative Verification Attributions with Keyword Fronting
Sui, Y., Ren, J., Tan, H., Chen, H., Li, Z., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70365-2_15 · Cited by 3
Faithful Self-Refinement in Mathematical Reasoning via Progressive Back-Translation
Liao, H., Zhu, Z., Hu, S., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890054 · Cited by 3
1-PAGER: One Pass Answer Generation and Evidence Retrieval
Jain, P., Soares, L. B., & Kwiatkowski, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.967 · Cited by 5
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
Towards Generative Search and Recommendation: A keynote at RecSys 2023
Chua, T. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3642986 · Cited by 3
“Additionally, Gao et al. [2023a] and Zhao et al. [2023] leverage trusted websites for addressing verification questions and performing trustworthy edits on the LLM-generated content.”
BackgroundCorrecting Factual Errors in LLMs via Inference Paths Based on Knowledge Graph
Ye, W., Zhang, Q., Zhou, X., Hu, W., Tian, C., & Cheng, J. (2024) · International Conference on Computational Linguistics and Natural Language Processing (CLNLP) · doi:10.1109/clnlp64123.2024.00011 · Cited by 4
“For instance, methods like RARR [12], VE [13], and FacTool [14] retrieve external knowledge after generation for post-editing.”
MethodXu, T., Piao, Z. H., Mukai, T., Murayama, Y., & Izumi, K. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200496 · Cited by 4
GraphSynth: Synthesis of Network Configuration Templates Using Large Language Models
Zhang, X. X., Gao, X., Tao, P., & Feng, T. (2025) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1145/3728725.3728742 · Cited by 4
Atomic Fact Decomposition Helps Attributed Question Answering
Yan, Z., Wang, J., Chen, J., Li, X., Liang, J., Li, R., & Pan, J. Z. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3608716 · Cited by 14
“RARR [9] is the first framework to implement question decomposition-based retrieval followed by revision.”
InfluentialXia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 3
Attribute or Abstain: Large Language Models as Long Document Assistants
Buchmann, J., Liu, X., & Gurevych, I. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.463 · Cited by 3
On the Capacity of Citation Generation by Large Language Models
Qian, H., Fan, Y., Zhang, R., & Guo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_9 · Cited by 6
Self-citationMDA: a multi-agent framework for data analysis task
Zhang, X., Kong, H., & Gao, S. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11369-1 · Cited by 2
REVISE: A Framework for Paragraph-Level Misinformation Correction in Large Language Models
Tripathi, S., Jennifer, T., Griffith, H., & Rathore, H. (2025) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest66680.2025.00027 · Cited by 2
“Unlike the previous approach [5], which revises each sentence independently without awareness of earlier changes, our method incorporates the history of prior revisions directly into the LLM’s input.”
InfluentialUnifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 14
“Two common approaches to implementing v TE are human reasoning [7, 18] and automated systems capable of natural language inference (NLI) [21, 22].”
BackgroundMethodInfluentialApplication Analysis of Generative Artificial Intelligence in The field of Open Source Intelligence
Zhou, L., Qin, Y., Yan, S., Zhang, G., & Hu, W. (2024) · Youth Academic Annual Conference of Chinese Association of Automation (YAC) · doi:10.1109/yac63405.2024.10598766 · Cited by 2
Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
Onweller, H., Lumer, E., Huber, A., Ramchandani, P., Subbiah, V. K., & Feld, C. (2026) · arXiv · doi:10.48550/arxiv.2605.06635 · Cited by 2
Retrieval-Augmented Multilingual Citation Generation
Liang, X., Niu, S., Zhang, S., Li, Z., Zhang, X., Wu, B., Xiong, F., Tang, B., Wang, H., Song, S., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888712 · Cited by 1
“[26] not only automatically identifies external evidence for the outputs of language models but also conducts post-editing to rectify hallucinated content present in the original response.”
PoliChat: Retrieval Augmented Generation on University Documents and Regulations
Wojtasik, K., Berdowski, A., Okulska, I., & Piasecki, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97570-7_21 · Cited by 2
A Hybrid Neuro-Symbolic Reasoning Framework for Automated Factual Verification in Conversational AI
Chin, W. H., Guo, Y., Loo, C. K., et al. (2025) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc66258.2025.11280070 · Cited by 1
SpecFSM: Extracting and Repairing Finite State Machines from Protocol Specification Documents
Li, X., Huang, D., Guan, J., & Li, H. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834331
Medicherla, C., Kulkarni, C., Patel, T., Shivam, S., Nutalapati, V., Parikh, M., Kumari, M., & Elengovan, A. K. (2026) · International Seminar on Intelligent Business and Edge-Computing Research (ISIBER) · doi:10.1109/isiber68248.2026.11470068 · Cited by 2
“Supervised methods like RARR [12] and FActScore [13] achieve high accuracy (F1=0.85-0.90) through fine-tuned classifiers but require 1,000+ labeled examples per domain.”
InfluentialSelf-Auditing LLMs for Bias and Hallucination Reduction Through Multi-Agent Frameworks
Anointina, Bamini, A. M. A., & Chitra, R. (2026) · International Conference on Expert Clouds and Applications (ICOECA) · doi:10.1109/icoeca68095.2026.11485555 · Cited by 2
“RARR introduced a retrieve-and-revise framework in which LLMs research evidence and correct unsupported claims [5].”
Detecting Hallucinations in Arabic Large Language Models Using Automatic Fact-Verification Signals
Alhawarat, M. O. (2026) · International Conference on Emerging Technologies and Engineering Systems (ICETES) · doi:10.1109/icetes68504.2026.11518602 · Cited by 1
“This was widened to fact verification - finding Wikipedia evidence to back claims [35].”
Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering
Shi, Y., Sun, M., Liu, Z., Yang, M., Fang, Y., Sun, T., & Gu, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3794658 · Cited by 19
Enhancing Asymmetric Web Search through Question-Answer Generation and Ranking
Ye, D., Liu, J., Fan, J., Tian, B., Zhou, T., Chen, X., & Ma, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671517 · Cited by 1
Investigating ChatGPT Translation Hallucination from an Embodied-Cognitive Translatology Perspective
Jiao, H., Li, X., Ding, J., & Xiao-jun, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2885-8_12 · Cited by 1
RARR Unraveled: Component-Level Insights into Hallucination Detection and Mitigation
Ross, J. J., Khramtsova, E., van der Vegt, A. H., Koopman, B., & Zuccon, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730337 · Cited by 3
Smaller Large Language Models Can Do Moral Self-Correction
Liu, G., Xue, Z., Zhang, X., Wang, R., & Johnson, K. (2025) · Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025) · doi:10.18653/v1/2025.trustnlp-main.5 · Cited by 1
TopoRefine: Iterative Refinement with Reasoning Topology as High-Level Feedback
Liao, H., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10888346
Nam, Y., Jeong, J., & Kim, N. (2026) · arXiv · doi:10.48550/arxiv.2606.09500 · Cited by 4
Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
Caruzzo, C., Yoo, D., & Kim, T. S. (2026) · arXiv · doi:10.48550/arxiv.2607.09349
“…[Maynez et al., 2020, Ji et al., 2023, Huang et al., 2025] and that RAG responses accurately reflect retrieved content with genuine citations [Gao et al., 2023, Asai et al., 2024, Es et al., 2024, Min et al., 2023, Kamoi et al., 2023, Stolfo, 2024, Saad-Falcon et al., 2024, Dziri et al.,…”
Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K. (2026) · arXiv · doi:10.48550/arxiv.2607.08700
“These systems produce long-form answers where each factual claim is supported by a retrieved, cited source, and faithful citation generation of this kind is an active RLVR training target [Nakano et al., 2022, Menick et al., 2022, Asai et al., 2024, Gao et al., 2023b, Bohnet et al., 2022].”
KIS-SQL: Knowledge-Enhanced In-Context Learning and Self-correction of Text-to-SQL
Fan, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7184-4_11
A Claim Decomposition Benchmark for Long-Form Answer Verification
Zhang, Z., Fan, Y., Zhang, R., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_4
Self-citation
Metadata
- Authors
- Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., & Guu, K.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- University of California, Irvine (US)
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 16477-16508
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385570481
- Cited by
- 141
- References
- 57
- Citations by year
- 2026: 61 · 2025: 43 · 2024: 21 · 2023: 16
- FWCI
- 25.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- language models, large language models, research question formulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
