The DOI matches its Crossref record; 60 of the 63 references it cites were found in a registry or an index; 28 of the 31 sources its claims cite were checked for support.
RARR: Researching and Revising What Language Models Say, Using Language Models
Open accessAbstract
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
RARR is a model-agnostic research-and-revise system that retrieves web evidence for an LM-generated passage and post-edits unsupported content while preserving the original text. Across factoid QA, reasoning chains, and dialog, RARR improves attribution more robustly than prior edit models and preserves original intent over 90% of the time.
Key findings
- On PaLM NQ outputs, RARR raises auto-AIS from 45.6 to 54.9 and human AIS from 35.4 to 43.4, with intent preservation 90.0, Levenshtein preservation 89.6, combined preservation 83.1, and F1_AP 57.0.
- On PaLM SQA outputs, RARR raises auto-AIS from 37.6 to 45.1 and AIS from 24.5 to 31.5, with intent preservation 92.6, Levenshtein 89.9, combined 84.6, and F1_AP 45.9.
- On LaMDA QReCC outputs, RARR raises auto-AIS from 18.8 to 29.4 and AIS from 13.2 to 28.3, with intent preservation 95.6, Levenshtein 80.2, combined 78.1, and F1_AP 41.5.
- RARR is the only method reported to perform robustly across all three datasets; EFEC and LaMDA preserve original intent only 6–40% of the time, while RARR preserves intent over 90% of the time. Editing increases attribution by up to 13% absolute while changing only 10–20% of the text.
Methods
- The paper formalizes Editing for Attribution: given a generated passage x, output a revised passage y plus an attribution report A of evidence snippets. It evaluates attribution with sentence-level AIS and an automated NLI-based auto-AIS, and preservation with human intent preservation, Levenshtein similarity, and their combination.
- RARR uses few-shot prompting with PaLM 540B: comprehensive question generation, Google Search retrieval (K=5 pages, top J=1 evidence per query), an agreement model to detect disagreement, an edit model to revise only when needed, and selection of at most M=5 evidence snippets for the attribution report.
- Experiments use 150 development and 150 test passages for each generation-model/source-dataset combination: PaLM and GPT-3 on NQ and SQA, and LaMDA and GPT-3 on QReCC. Baselines include EFEC and LaMDA Research; human evaluation and automated metrics are both reported.
Limitations
- Stated: attribution and preservation may not deserve equal weight depending on the application; the metrics do not capture all aspects of attribution, such as self-evident sentences, varying assertion scope, directness, or implicatures; preservation metrics do not cover discourse or logical coherence, and heavily flawed inputs may require changes that the metrics penalize.
- Stated: RARR is not fully general—it is not prepared for poetry or long documents without prompt changes—and it tends to preserve rather than delete claims it cannot attribute, even when those are hallucinations. It is also computationally costly because it prompts a large language model.
- Evident from the text: the main experiments are based on a single run, though Appendix A reports variance over 3 random runs; the paper also notes that evidence trustworthiness and conflicting evidence are not addressed by the current implementation or evaluation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
49 Own result · 18 Own interpretation · 18 No basis given · 12 Supported · 10 Own method · 8 Partly supported · 1 Sources not found · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We utilize the natural language inference (NLI) model from Honovich et al. (2022), which correlates well with AIS scores.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
What the sources say
- Partly supportedcites Honovich et al. (2022) sourceExperimental — accuracy not yet measured
Section 3.3 and Appendix B describe an NLI model trained by Honovich et al. (2022), supporting the model identification. Section 4 reports its performance using ROC AUC on factual-consistency datasets, but the copy does not report a correlation with AIS scores (searched: AIS scores, Attributable to Identified Sources). (LLM, conf=0.86)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite these incredible advances, state-of-theart LMs still frequently produce biased, misleading, or unsupported content, colloquially called”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Partly supportedcites Maynez et al., 2020 sourceExperimental — accuracy not yet measured
Section 2.1 defines hallucinations as summary content unsupported by or misrepresenting the source document, and the assessment in Section 5.2 finds substantial hallucinations in the summarization systems studied. The paper does not establish that language models generally frequently produce biased content; its findings concern abstractive summarization. (LLM, conf=0.87)
- Partly supportedcites Menick et al., 2022 sourceExperimental — accuracy not yet measured
The opening passage (p. 1) says large language models can hallucinate convincing nonsense, and the Conclusion (p. 18) says they often produce hallucinated facts. The Limitations section (p. 18) discusses misleading, cherry-picked quotations. The paper does not establish that language models frequently produce biased content; its discussion of bias concerns supporting sources such as Wikipedia. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Even retrievalaugmented models (Guu et al., 2020;Lewis et al., 2020), which retrieve relevant documents and then condition on them to generate text, still do not guarantee attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Supportedcites Guu et al., 2020 source
Appendix C (p. 11) reports that REALM can retain facts in its parameters and make predictions that do not update when the retrieved corpus changes. This supports the claim that retrieval alone does not guarantee a prediction is attributable to the retrieved documents. (LLM, conf=0.88)
- Supportedcites Lewis et al., 2020 source
The Introduction (p. 2) describes RAG as retrieving documents and conditioning generation on them. But H Retrieval Collapse (p. 19) reports cases where the generator learned to ignore those documents, supporting the claim that retrieval does not guarantee attribution to them. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In real-world scenarios such as the one which RARR operates in, relevant evidence may not be provided, necessitating retrieval (Fan et al., 2020;Piktus et al., 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Fan et al., 2020 source
Section 3.2 (Question Answering; page not specified) says crowdworkers receive a claim, its source, and a question, then search the web for information to answer it. Section 4 (QABrief Model; page not specified) likewise retrieves an evidence document for each question. These passages support the claim that relevant evidence must be retrieved rather than being provided at the outset. (LLM, conf=0.90)
- Supportedcites Piktus et al., 2021 source
Section 2.1 (p. 3) says knowledge-intensive tasks require access to an external knowledge source and are typically solved by retrieving relevant documents before generating an answer. This supports the cited point that relevant evidence may need to be retrieved rather than supplied with the task. (LLM, conf=0.90)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even if a revision preserves intent, it may still make superfluous modifications, such as reordering words, changing textual style, or including unnecessary additional information (Thorne and Vlachos, 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.2 Measuring preservation
What the sources say
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
Section R2—Supported by Evidence says corrections should target errors in the original claim rather than introduce unrelated information; Section 7 Evaluation also checks whether unrelated information was added. These passages support the concern about unnecessary additions, but the copy does not discuss intent-preserving revisions that needlessly reorder words or change textual style. No page numbers are available in the indexed passages. (LLM, conf=0.83)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Evidence-based Factual Error Correction (EFEC; Thorne and Vlachos, 2021) also implements a full research-and-revise workflow trained on Wikipedia passages (Thorne et al., 2018).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Thorne and Vlachos, 2021 source
The Retrieved Evidence section says EFEC retrieves evidence from Wikipedia and rewrites claims to better match it. Section 6 (Data) identifies the training basis as FEVER claims paired with Wikipedia evidence, supporting the description of a research-and-revise workflow trained on Wikipedia passages. Page numbers are unavailable in our copy. (LLM, conf=0.90)
- Partly supportedcites Thorne et al., 2018 sourceExperimental — accuracy not yet measured
The Introduction and §3.1 (page numbers unavailable) establish that FEVER’s claims and verification evidence come from Wikipedia. The cited paper does not address EFEC or establish that EFEC was trained on those passages. The research-and-revise workflow is attributed to the separate 2021 citation, not this one. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In this paper, we generally avoid the term”
- Also stated in: § E Details on the dataset · plain
“Even if a claim is attributed to a particular source, it does not guarantee that the source is”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
What the sources say
- Supportedcites Menick et al., 2022 source
Section 3.7 (pp. 16–17) distinguishes a claim being supported by evidence from its being true. It reports examples of false claims rated as supported and concludes that a high support score can coexist with a low truthfulness score. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § E Details on the dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we find that existing revision models do not always generalize across many tasks (and were not originally intended to), and therefore propose a new research-and-revise model that leverages the power of few-shot prompting in large language models to robustly generalize across domains.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By manually inspecting 30 examples from our benchmarks, we found M = 5 snippets to be sufficient for full attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only RARR has a robust F1 AP across all tasks.”
- Also stated in: § E Details on the dataset · plain
“Only RARR has a robust F1 AP across all tasks.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § E Details on the dataset · plain
- It may state two separate claims as one. 31
- Probably not a substantial claim. 36
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prior work has shown that retrieval-augmented models generate text that either includes additional information outside the retrieved documents (Dziri et al., 2022), ignores the documents altogether (Krishna et al., 2021), or even contradicts the documents (Longpre et al., 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Supportedcites Dziri et al., 2022 source
The Dziri citation is attached only to the claim that generated text includes information outside the retrieved documents, not to the later claims about ignoring or contradicting them. The Introduction reports that model-generated responses contain hallucinations, including unsupported objective factual information; Section 3 reports that the models generate and amplify hallucinations relative to their source knowledge. The paper describes these as knowledge-grounded conversational models using knowledge snippets rather than using the citing sentence’s term “retrieval-augmented.” (LLM, conf=0.90)
- Supportedcites Krishna et al., 2021 source
The Introduction states that the authors’ retrieval-augmented system does not actually use the documents it retrieves. Section 3.1 supports this with a control experiment: replacing predicted retrievals with random documents had almost no effect on generated answers. The Krishna et al. citation is attached specifically to the claim that models ignore retrieved documents, not to the sentence’s other two claims. (LLM, conf=0.94)
- Supportedcites Longpre et al., 2021 source
The Results passage under §3.3 Metrics reports that, when the provided passage contained a substituted answer, generative QA models sometimes predicted the original answer learned during training instead, contradicting the provided passage. The Longpre et al. citation is attached only to the claim about contradicting documents, not the other two behaviors in the sentence. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“LaMDA (Thoppilan et al., 2022) generates responses in three steps: 1) generate a “base response”; 2) generate search queries from the base response; 3) generate a “revised response” conditioned on the base response and retrieved evidence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Thoppilan et al., 2022 source
The fine-tuning description (p. 8) says LaMDA takes a response from the base model, generates a toolset query, and predicts a grounded response from the base response, query, and retrieved snippet. The example (p. 9) calls these the Base and Research phases; the Research phase can issue multiple queries before responding to the user. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“EFEC uses a neural retrieval model (Karpukhin et al., 2020) to retrieve from Wikipedia”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Karpukhin et al., 2020 source
The cited paper describes a neural dense-passage retriever: §3.1 says it uses BERT encoders to retrieve passages, and §4.1 identifies those passages as coming from Wikipedia. The citation identifies the retrieval model used by EFEC, rather than asking this source to document EFEC itself. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FRUIT (Logan IV et al., 2022) and PEER (Schick et al., 2022) both implement an editor that is fine-tuned on Wikipedia edit history with the goal of updating outdated information and collaborative writing respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Not matchedcites Logan IV et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Schick et al., 2022 source
The Schick et al. citation attaches to the PEER half of the sentence, not the FRUIT half. The Introduction (p. 2) describes PEER’s goal as collaborative editing, while Training Data (p. 5) and Experimental Setup (p. 6) say it was trained from Wikipedia edit history starting from a pretrained model. The paper identifies FRUIT separately as a dataset for updating Wikipedia texts (Downstream Tasks, p. 8). (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Language models can generate reasoning chains to answer complex questions (Wei et al., 2022).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.1 Evaluation setups
What the sources say
- Supportedcites Wei et al., 2022 source
Section 2 (p. 2) states that sufficiently large language models can generate chains of thought when given few-shot demonstrations, and gives a multi-step math question as an example. The claim says models can do this, not that all models do so without prompting. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A similar strategy has been employed to train textplanning models (Narayan et al., 2022).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Query generation
What the sources say
- Supportedcites Narayan et al., 2022 source
The Introduction says the authors use question-generation technology to create question–answer blueprints that serve as text plans. The Iterative Model section describes training a model to produce those blueprints while planning and generating text one sentence at a time. This supports the citing sentence’s characterization of a similar strategy used to train text-planning models. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previously, Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
What the sources say
- Supportedcites Rashkin et al. (2021) source
The Introduction (p. 2) explicitly introduces AIS as the authors’ evaluation framework. Definition 8 (p. 10) defines whether a sentence is AIS using conditions that either hold or do not, supporting the description of a binary notion of attribution; the paper also reports human evaluation studies in the Conclusion (p. 26). (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The approach is inspired by works on fact-checking 2 where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans (Thorne et al., 2018;Schuster et al., 2021;Thorne and Vlachos, 2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Partly supportedcites Thorne et al., 2018 sourceExperimental — accuracy not yet measured
Section 3.2 (Task 2—Claim Labeling; page not supplied) describes annotators finding Wikipedia evidence to support or refute human-generated claims. This supports citing FEVER as fact-checking work, but the paper does not establish that a research-and-revise workflow is effective at attributing or correcting unattributed human claims. (LLM, conf=0.82)
- Partly supportedcites Schuster et al., 2021 sourceExperimental — accuracy not yet measured
Section 4.1 (page unavailable) says Wikipedia users verify and correct others’ revisions, and §3.1 describes collecting factual edits. The paper does not establish that simple research-and-revise workflows are effective, or address attributing previously unattributed human claims. (LLM, conf=0.82)
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
The Retrieved Evidence section reports that the system retrieves evidence and uses it to correct claims, with satisfactory corrections for some evaluated instances. The Conclusions and Future Work section says the evaluation used mutations introduced by human annotators and may not represent errors in real-world documents. The paper does not demonstrate attribution of unattributed claims, so that part and the broader implication about human-made claims are not supported; page numbers are unavailable in our copy. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“EFEC fine-tunes a T5-based model to revise text conditioned on multiple evidence snippets using both semi-supervised and fully-supervised approaches.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Partly supportedcites Thorne and Vlachos, 2021 sourceExperimental — accuracy not yet measured
Section 5.3 says the paper fine-tunes a T5 model to generate corrections from masked claims and evidence, and Section 4.1 specifies multiple retrieved evidence sentences. The paper calls its main training approach “distant supervision,” not semi-supervision; it also fine-tunes a separate fully supervised T5 model as a performance ceiling (Implementation, “Fully Supervised Ceiling”). (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We compare against their fully-supervised approach, which performed best in their experiments.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
What the sources say
- Supportedcites Thorne and Vlachos, 2021 source
The Results section states that the fully supervised models had the highest rate of satisfactory corrections that improved factuality. It also reports 68.9% satisfactory corrections with gold evidence, supporting the citing sentence’s comparison. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This form of question-guided agreement was previously explored by Honovich et al. (2021).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.2 Revision stage
What the sources say
- Supportedcites Honovich et al., 2021 source
Figure 2 describes generating questions from a response, answering them using grounding knowledge, and comparing the answers. The “Answer Similarity and Final Scores” section explains how those comparisons assess agreement between the response and the knowledge, supporting the paraphrase. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“An exception is LaMDA (Thoppilan et al., 2022), a language model for dialog that performs revision by training on human annotations.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Supportedcites Thoppilan et al., 2022 source
The Abstract (p. 1) identifies LaMDA as a language model specialized for dialog. Section 5 (p. 6) says crowdworkers edit model responses, and Section 6.2 (p. 8) says their corrections provide fine-tuning data that LaMDA learns to imitate. This supports the paraphrase that it learns to revise responses from human annotations. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FRUIT (Logan IV et al., 2022) and PEER (Schick et al., 2022) both implement an editor that is fine-tuned on Wikipedia edit history with the goal of updating outdated information and collaborative writing respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.3 Attribution report
What the sources say
- Not matchedcites Logan IV et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Schick et al., 2022 source
The Introduction (p. 2) describes PEER’s goal as a collaborative writing model. Training Data (p. 5) says its main training data comes from Wikipedia’s edit history, and Experimental Setup (p. 6) says PEER is initialized from LM-Adapted T5 and trained on approximately five million Wikipedia edits. Section 4 (p. 4) identifies PEER-Edit as the main model for editing. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results on GPT-3 passages show similar trends (Appendix A).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this case, RARR using CQGen needs to sacrifice more preservation score to edit the text, leading to a lower F1 AP score.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6.2 Ablations
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of F1 AP , RARR is the only method that performs robustly across all three datasets, and significantly outperforms prior methods on NQ and SQA.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The attribution performance of sentences-as-queries drops significantly, while CQGen is more robust.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6.2 Ablations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The answer tends to be context-dependent, featuring pronouns and implicit references.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Knowledge-intensive dialogs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After editing, RARR can increase attribution by up to 13% absolute, while changing only 10-20% of the text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, more revision leads to less preservation score and spurious changes to the text passage, as demonstrated in Figure 8.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Ablating agreement model
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that RARR not only preserves the short answer accuracy but actually improves it by roughly 5%.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact on downstream task performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RARR significantly improves attribution while preserving most of the original text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in datasets where the claim contains multiple sentences (NQ and SQA), such a behavior yields low preservation scores, and also results in outputs that are less informative.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“however, not all passages in our experiments are supported by Wikipedia articles.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“EFEC reduces the passage x into a single sentence.”
- Also stated in: § E Details on the dataset · plain
“EFEC reduces the passage x into a single sentence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § E Details on the dataset · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LaMDA changes the writing style.”
- Also stated in: § E Details on the dataset · plain
“LaMDA changes the writing style.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Also stated in: § E Details on the dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EFEC frequently attempts to summarize the entire passage into one sentence, or drops later sentences.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also see that editing is crucial to improve attribution: if we only retrieve evidence to support the original response x without editing, attribution ranges from the low 10s to mid 30s.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting passages are mostly coherent but often contain factual errors.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.1 Evaluation setups
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we do not provide a means for judging trustworthiness, the design of RARR does allow for the research stage to restrict its search over a userspecified corpus, based on what the user deems trustworthy.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, RARR tends to preserve rather than delete claims that it cannot attribute.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that RARR's performance is low on examples with mathematical reasoning, as these are beyond the capability of the edit model with our current prompt.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sometimes the whole output is based on a false premise and needs to be completely rewritten, in which case RARR cannot satisfactorily edit due to our revision threshold (Section 3.2).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the AIS scores from human correlate well with auto-AIS scores, with some bias for non-attributed sentences to be judged as attributed by auto-AIS.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Comparing human and automated evaluation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When RARR is not 100% successful in making text consistent with retrieved evidence, the revised text will be partially attributed.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, our model is computationally costly, since it is based on prompting a large language model.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The editor occasionally produces large edits that bring the new revision close to e but far from the current y.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EFEC and LaMDA also face this issue, but instead trade preservation for attribution and rewrite the text to a different topic.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“proves attribution of the explanations on all four categories of MMLU, although the increases are relatively small.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conducted correlation studies between human and automatic metrics and found strong Pearson correlation (attribution = 0.74; preservation = 0.62).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Comparing human and automated evaluation
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This work does not address the question of whether a source is trustworthy, or the related topic of misinformation.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The standard deviations of Attr auto , Pres Lev , and F1 AP are 1.2, 0.5, and 1.0 respectively.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > A Additional experiments and analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In fact, occasionally ignoring the retrievals can make the models more robust to bad retrievals (Khandelwal et al., 2020), illustrating that end-task performance and attribution are not always aligned.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 1 Introduction
What the sources say
- Not matchedcites Khandelwal et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The main experiments in Section 5 are based on a single run.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > A Additional experiments and analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This helps reduce the editor's deviation from the current y.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 3.2 Revision stage
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This behavior generally increases the attribution score, because it is usually easier to make one sentence fully attributable than many sentences.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 6 Analysis 6.1 Qualitative analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- Rests on an assumption: Most existing LMs lack a built-in mechanism for attribution.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We suspect that this may be due to noisy retrievals, which sometimes provide misleading evidence (exemplified in Figure 7d).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact on downstream task performance
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To more fairly compare the editing capabilities of EFEC, we instead use the evidence retrieved by our research stages (CQGen and web search).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.2 Models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The authors have manually looked through most of the data and found no personal identifiers.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In the main paper, we present two automatic metrics to conveniently gauge these properties, but rely on a human evaluation as the gold standard.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > C Details on human evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We reject an editing if the edit distance is more than 50 characters or more than half of the original text length.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We had 3 annotators rate each example in the pilot phase to measure inter-annotator agreement, and had a single rater annotate each example afterwards.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Rater recruitment and training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We tune our hyperparameters on the validation set as well.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These news article summaries are often about less widely known people and events, which is challenging for retrievers, leading to low attribution.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result suggests that using web search with standard question generation methods may fail to capture important context from the input, and is not sufficient for the attribution task.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Challenging domains
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We manually wrote our prompts by eye-balling quality on a dozen of examples from a separate validation set.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Most existing LMs lack a built-in mechanism for attribution.
“Most existing LMs, such as those based on sequence-to-sequence architectures, lack a builtin mechanism for attribution.”
The paper relies on this without support.
Pres intent requires human annotation.
“Since Pres intent requires human annotation, we use Pres Lev as an automated metric for model development.”
The paper relies on this without support.
Gold or reference edits are often only available for specialized domains.
“which are often only available for specialized domains.”
The paper relies on this without support.
After retrieving evidence, some parts of the passage may be properly attributed, but other parts remain unattributed and should be revised.
“After retrieving evidence, certain parts of x may now be properly attributed, but other parts remain unattributed and should be revised.”
The paper relies on this without support.
A prompt with six human demonstrations is sufficient for PaLM to adequately learn the comprehensive question generation task.
“A prompt with six human demonstrations was sufficient for PaLM to adequately learn the task.”
The paper relies on this without support.
LaMDA was not optimized for preservation.
“since LaMDA was not optimized for preservation.”
The paper relies on this without support.
EFEC's training data was limited to single sentences.
“which was limited to single sentences.”
The paper relies on this without support.
The 40 manually inspected examples are representative of the question generation model's coverage.
“a manual inspection of 40 examples shows > 80% with questions that fully cover all aspects of the input text.”
The authors state this as an assumption.
Manually editing examples until judged 100% attributable approximates the minimal amount of editing needed to make a text passage fully attributed.
“As an approximation, we manually edited 30 examples in our NQ benchmark until we judged them to be 100% attributable.”
The paper relies on this without support.
Attribution and preservation are both subjective properties that may change with even small edits.
“Attribution and preservation are both subjective properties that may change with even small edits.”
The paper relies on this without support.
There is currently no linguistic consensus on what constitutes a claim.
“While each sentence may contain multiple claims that could be attributed independently, there is currently no linguistic consensus on what constitutes a claim.”
The paper relies on this without support.
Utterances can make assertions with varying levels of directness.
“In addition to varying scope, utterances can also make assertions with varying levels of directness.”
The paper relies on this without support.
Some content may be simultaneously supported by certain sources, while contradicted by others.
“There is also the possibility that some content may be simultaneously supported by certain sources, while contradicted by others.”
The paper relies on this without support.
Few-shot prompting has proven to be effective for many recent large language models.
“Few-shot prompting has proven to be effective for many recent large language models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, we can do this without a separate claim decomposition step.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Gao et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Gao et al. (2023); Min et al. (2023); Wang et al. (2023) decompose each generated sentence into atomic facts and then search for relevant documents to support each atomic fact.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Gao et al. (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent studies (Gao et al. 2023; Chern et al. 2023) highlight the potential of LLMs to detect factual content in texts.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Gao et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“For example, methods that use search engines only for self-correction (Gao et al., 2023; Yu et al., 2023) are unfair because they can use search engines to directly improve the initial response generation.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“A popular source of knowledge is search engines, which are often used with queries generated from initial responses to retrieve information for validating their correctness (Gao et al., 2023;Jiang et al., 2023b).”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“As shown in Table 5, some prior studies (Gao et al., 2023;Yu et al., 2023;Zhao et al., 2023) use external knowledge only for selfcorrection, while they can also directly use external knowledge to improve the initial response generation process.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
In the citing paper
“For example, RARR (Gao et al., 2023) uses external knowledge to detect mistakes in initial responses, while it does not use any external knowledge when generating initial responses.”
From When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (2024) · cites this paper as Gao et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
“After editing, RARR can increase attribution by up to 13% absolute, while changing only 10-20% of the text.”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model auto-AIS AIS intent Lev comb F1 AP
“RARR 18.8 → 29.4 13.2 → 28.3 95.6 80.2 78.1 41.5”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 5.3 Results
“We found that RARR is the only method that preserves the original intent of x over 90% of the time”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Model auto-AIS AIS intent Lev comb F1 AP
“RARR 45.6 → 54.9 35.4 → 43.4 90.0 89.6 83.1 57.0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > Impact of model scale
“We also experimented with keeping the editor stage at 540B while shrinking the query generation stage to 64B”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > E Details on the dataset
“Full RARR qgen 62B, editor 540B”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 9 Ethical considerations
“One could identify unattributed parts using either the automated attribution score (Attr AIS ) or the relevance scores used to generate the attribution report (Section 3.3).”
§ RARR: Researching and Revising What Language Models Say, Using Language Models > 2.1 Measuring attribution
“During model development, we define an automated metric, auto-AIS (Attr auto ), that approximates human AIS judgments.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection
Wang, W. Y. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2067 · Cited by 2,010
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Self-citationLatent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771
Large Dual Encoders Are Generalizable RetrieversPreprint
Ni, J., Qu, C., Lu, J., Dai, Z., Abrego, G. H., Ma, J., Zhao, V., Luan, Y., Hall, K. B., Chang, M.-W., et al. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2112.07899 · Cited by 715
Self-citationEvaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
Self-citation(2021) · International Conference on Learning Representations · Cited by 435
SciTaiL: A Textual Entailment Dataset from Science Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12022 · Cited by 419
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319
PAWS: Paraphrase Adversaries from Word Scrambling
Zhang, Y., Baldridge, J., & He, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1131 · Cited by 306
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
Augenstein, I., Lioma, C., Wang, D., Lima, L. C., Hansen, C., Hansen, C., & Simonsen, J. G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1475 · Cited by 226
(2022) · doi:10.18653/v1/2022.emnlp-main.669 · Cited by 177
Self-citationShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144
Constrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139 · cited 2×
“, 2022), use tools and APIs (Shin et al., 2021; Thoppilan et al., 2022), and answer open-domain questions (Petroni et al.”
BackgroundFactual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
Open-Domain Question Answering Goes Conversational via Question Rewriting
Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., & Chappidi, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.44 · Cited by 130
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · cited 3×
“This form of question-guided agreement was previously explored by Honovich et al. (2021).”
BackgroundMethodRetrieval-Based Neural Code Generation
Hayati, S. A., Olivier, R., Avvaru, P., Yin, P., Tomasic, A., & Neubig, G. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1111 · Cited by 103
Entity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101
Self-citationFully Automated Fact Checking Using External Sources
Karadzhov, G., Nakov, P., Màrquez, L., Barrón‐Cedeño, A., & Koychev, I. (2017) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-049-6_046 · Cited by 100
Annotating and Modeling Fine-grained Factuality in Summarization
Goyal, T., & Durrett, G. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.114 · Cited by 100
Neural Data Augmentation via Example ExtrapolationPreprint
Lee, K., Guu, K., He, L., Dozat, T., & Chung, H. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01335 · Cited by 57
Self-citationTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
Decontextualization: Making Sentences Stand-Alone
Choi, E., Palomaki, J., Lamm, M., Kwiatkowski, T., Das, D., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00377 · Cited by 54
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 4×
““hallucinations”) (Maynez et al., 2020; Menick et al., 2022).”
BackgroundInfluentialPEER: A Collaborative Language Model
Schick, T., Dwivedi-Yu, J., Jiang, Z., Petroni, F., Lewis, P., Izacard, G., You, Q., Nalmpantis, C., Grave, E., & Riedel, S. (2022) · arXiv · doi:10.48550/arxiv.2208.11663 · Cited by 49
Generating fact checking briefs
Fan, A., Piktus, A., Petroni, F., Wenzek, G., Saeidi, M., Vlachos, A., Bordes, A., & Riedel, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.580 · Cited by 42
Evidence-based Factual Error Correction
Thorne, J., & Vlachos, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.256 · Cited by 39
Automatic Fact-Guided Sentence Modification
Shah, D., Schuster, T., & Barzilay, R. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i05.6406 · Cited by 36
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2112.12870 · Cited by 34
Balachandran, V., Hajishirzi, H., Cohen, W. W., & Tsvetkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 33
Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
Bohnet, B., Tran, V. Q., Verga, P., Aharoni, R., Andor, D., Soares, L. B., Ciaramita, M., Eisenstein, J., Ganchev, K., Herzig, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.08037 · Cited by 30
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A. P., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00583 · Cited by 30
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus
Piktus, A., Petroni, F., Karpukhin, V., Okhonko, D., Broscheit, S., Izacard, G., Lewis, P., Oğuz, B., Grave, E., Yih, W.-T., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09924 · Cited by 22
FRUIT: Faithfully Reflecting Updated Information in Text
Iv, R., Passos, A., Singh, S., & Chang, M.-W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.269 · Cited by 19
Iso, H., Qiao, C., & Li, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.17 · Cited by 17
Research works printed but not linked: 1
- Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Gra- ham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In NAACL. · DOI
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“To enhance the trustworthiness and attribution of LLMs, Gao et al. (2023a) adopt the research-then-revise workflow, which initially research relevant evidence and subsequently revise the initial generation based on detected discrepancies with the evidence.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
“…to enhance information accuracy [143, 238], increasing model transparency to foster understanding of decision processes [86, 202, 203], developing fact-checking systems to verify outputs [200, 201], and designing hallucination detection tasks [325, 326] to evaluate and improve model performance.”
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282
Metadata
- Authors
- Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., & Guu, K.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- University of California, Irvine (US)
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 16477-16508
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385570481
- Cited by
- 141
- References
- 57
- Citations by year
- 2026: 61 · 2025: 43 · 2024: 21 · 2023: 16
- FWCI
- 25.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- language models, large language models, research question formulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
