VeriScore: Evaluating the factuality of verifiable claims in long-form text generation

Song, Yixiao; Kim, Yekyung; Iyyer, Mohit

doi:10.18653/v1/2024.findings-emnlp.552

Cited by 17Topic ModelingMisinformation and Its ImpactsNatural Language Processing Techniques

VeriScore: Evaluating the factuality of verifiable claims in long-form text generation

Open access

Song, Yixiao, Kim, Yekyung, Iyyer, Mohit·2024·Findings of the Association for Computational Linguistics: EMNLP 2024

DOI 10.18653/v1/2024.findings-emnlp.552

Open this paper in CoChat — free account Open PDF

Abstract

Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia.These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false).We address this issue with VERISCORE, 1 a metric for evaluating factuality in diverse long-form generation tasks that contain both verifiable and unverifiable content.VERISCORE can be effectively implemented with either closed or fine-tuned openweight language models.Human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks.We use VERISCORE to evaluate generations from 16 different models across multiple longform tasks and find that while GPT-4o is the best-performing model overall, open-weight models such as Mixtral-8 × 22 are closing the gap.We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., long-form QA), highlighting the need for expanding factuality evaluation across tasks with varying fact density.Claim 1: Beetroot gets its red color from betacyanin.Claim 2: Betacyanins are a type of anthocyanin.Claim 3: Anthocyanins are water-soluble pigments.Claim 4: Anthocyanins are commonly found in various fruits and vegetables.Claim 1: Beetroot gets its red color from betacyanin.Claim 2: Betacyanin is a special helper that gives beetroot its beautiful red color.Claim 3: Betacyanins are a type of anthocyanin.Claim 4: Anthocyanins are water-soluble pigments.Claim 5: Anthocyanins are commonly found in various fruits and vegetables Claim 6: Betacyanin is like a superhero cape.

AI snapshot

Written from the paper’s full text.

TL;DR

VeriScore is a factuality metric for long-form generation that extracts only verifiable claims, retrieves evidence via Google Search, and scores supported claims with F1@K. Across 16 LMs and multiple tasks, GPT-4o is best overall (avg 66.5) while open-weight Mixtral-8x22B is close (58.5), and task-specific factuality does not always correlate.

Key findings

Human evaluation preferred VeriScore’s extracted claims over SAFE’s 93% of the time; SAFE was preferred only 26 of 360 times, with 19 of those marginal, and extraction agreement was Fleiss κ = 0.7662. In claim verification, only 55% of claims were supported, 2.8% contradicted, and 42.2% inconclusive, with Fleiss κ = 0.7316.
Benchmarking 16 LMs on six long-form fact-seeking domains found GPT-4o best overall (avg 66.5), followed by GPT-4-0125-preview (65.5) and Claude-3-Opus (61.2); the best open-weight model was Mixtral-8x22B-Instruct-v0.1 (58.5), while Gemma-2B-it was worst (27.4).
Fine-tuned open-weight implementations were competitive: fine-tuned Mistral extraction had 43.7% exact match and RougeL 0.801 versus GPT-4, and fine-tuned Llama3 verification achieved F1 = 0.841 on human-annotated data. Evaluating 400 GPT-4o generations with the prompting method cost $1,038 USD.
LMs’ VeriScore on one task does not necessarily correlate with another; Kendall’s τ with average performance ranged from 0.56 to 0.83 across domains, supporting the need for multiple factuality tasks.

Methods

VeriScore pipeline: extract verifiable claims with sliding-window few-shot prompts, retrieve top ≤10 Google Search results via Serper, verify each claim as supported/contradicted/inconclusive (later collapsed to supported/unsupported), and compute F1@K where K is the median number of extracted facts per domain.
Human validation: pairwise extraction comparison on 15 texts from each of eight datasets (360 annotations, three raters), and a verification study on 320 GPT-4-extracted claims with three annotators.
Open-weight and benchmark setup: fine-tuned Llama3-8B-Instruct and Mistral-7B-Instruct-v0.2 on 13,403 verification data points and 99,592 extraction pairs; benchmarked 16 LMs on six long-form fact-seeking domains (LongFact, Biography, ELI5, AskHistorians, FreshBooks, ShareGPT) with 50 prompts per domain and max token length 1024.

Limitations

Stated: formally defining verifiable claims remains a working definition rather than a formal one; decomposition is slow (about 4 hours for 400 GPT-4o responses on one RTX8000, and about 2 hours to verify 10k claims); the extractor may still extract a small amount of unverifiable claims for extremely infactual outputs; and no exhaustive hyperparameter search was done for the fine-tuned extractor/verifier.
Evident from the text: Google Search snippets are often insufficient for complex claims requiring expertise or extensive reasoning, and unsupported claims frequently lack direct contradicting evidence rather than being clearly false.
Evident from the text: F1@K favors longer outputs; in FreshQA, GPT-4o had a higher supported-claim percentage (74.70%) than Claude 3 Haiku (72.71%) but a lower VeriScore because GPT-4o generated shorter outputs (3.39 claims per response vs. 5.5 for Claude 3 Haiku, with K = 4).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-emnlp.552

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

42 of 64 cited sources checked · 24 supported · 2 flagged · accuracy not yet measured

Interpretationreasoning
SAFE's generalizability is questionable given that it is only evaluated on FACTSCORE's biography data.
“Lastly, SAFE's generalizability is questionable given that it is only evaluated on FACTSCORE's biography data.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al., 2024 source
  Appendix C.10, “Generalization to more professional domains” (page unavailable), says the authors evaluated SAFE on law and medicine facts from six additional questions and reports agreement rates for both domains. The paper therefore contradicts the assertion that SAFE was evaluated only on FACTSCORE’s biography data. Whether those evaluations fully establish generalizability is a separate question. (LLM, conf=0.90)
  Checked by a reasoning model — may be wrong.
  Experimental — accuracy not yet measured
Premisecitation
Existing metrics FACTSCORE and SAFE decompose input text into atomic claims and verify each against a knowledge base like Wikipedia.
“Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into”
- Also stated in: § 1 Introduction · plain
  “Modern approaches for evaluating the factuality of LLM-generated long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), proceed in three stages: (1) decomposition of the text into a list of”
Quote in the paperSubstantial claim2 of 2 sources checked
- cites Min et al., 2023 source
- cites Wei et al., 2024 source
  The SAFE section and Appendix C.4 say that SAFE splits responses into individual facts and checks them against search results, using a splitting step almost identical to Min et al. (2023). The Related Work section says FActScore uses Wikipedia pages as knowledge sources. But SAFE verifies facts through iterative Google Search queries, not against a Wikipedia-like knowledge base, so that part of the claim does not accurately describe SAFE. (LLM, conf=0.87)
  Experimental — accuracy not yet measured
Premisecitation
FACTSCORE's extraction method does not resolve pronouns.
“First, it does not resolve pronouns: for example, it extracts”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Premisecitation
F1@K improves on measuring factual precision alone.
“F 1 @K (Wei et al., 2024) considers both factual precision and recall, which improves on measuring factual precision alone (Min et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Premisecitation
SAFE uses FACTSCORE's prompt without changes.
“First, besides adding a brief task description, SAFEuses FACTSCORE's prompt without changes.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al., 2024 source
  The ‘Split a response into individual facts’ section says SAFE uses the in-context exemplars from Min et al. (2023) and adds instructions of its own. Appendix C.4 calls this step ‘almost identical’ to Min et al.’s, but our copy does not establish that the rest of FACTSCORE’s prompt was used without changes or that the addition was only a brief task description. (LLM, conf=0.78)
  Experimental — accuracy not yet measured

Premisenone given
SAFE's revision and relevance check adds significant processing time and cost.
“Second, the revision and relevance check adds significant processing time and cost.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al., 2024 source
  The SAFE section (page unavailable) describes revision and relevance checking as steps in the pipeline, and Section 4 reports costs for the pipeline as a whole. It does not measure processing time or isolate the cost added by those two steps, so the claim that they add significant time and cost is not established. (LLM, conf=0.83)
  Experimental — accuracy not yet measured
Premisenone given
SAFE's relevance check unexpectedly removes verifiable claims.
“Third, the relevance check unexpectedly removes verifiable claims.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al., 2024 source
  The SAFE section says the relevance check filters facts before checking whether the remaining facts are supported. Appendix A.3 reports errors in relevance reasoning, but our copy does not establish that the check unexpectedly removes verifiable claims specifically; the paper describes excluding irrelevant facts as intentional. (LLM, conf=0.79)
  Experimental — accuracy not yet measured
Premisecitation
SAFE and FACTSCORE use closed LLMs for claim extraction and verification.
“SAFE and FACTSCORE use closed LLMs for claim extraction and verification.”
Quote in the paperNot a substantial claim2 of 2 sources checked
- cites Min et al., 2023 source
  For FACTSCORE, Appendix A.2 (p. 15) says InstructGPT is used to extract atomic facts, and Section 4.2 (p. 7) discusses ChatGPT as a verifier. But Section 4.2 also reports a LLAMA-based verifier and recommends trying both variants, so the claim overstates closed-LLM use for verification. This paper does not establish how SAFE works. (LLM, conf=0.88)
  Experimental — accuracy not yet measured
- cites Wei et al., 2024 source
  The SAFE section says an LLM extracts individual facts and checks them against search results; Appendix C.9 identifies its default LLM as GPT-3.5-Turbo. The Related work section says FActScore uses Wikipedia as a knowledge source for humans and language models rating biographies, but our copy does not establish that FActScore uses closed LLMs for both claim extraction and verification. (LLM, conf=0.84)
  Experimental — accuracy not yet measured
Premisecitation
Users rely on the accuracy of LLM-generated content.
“Users rely on the accuracy of LLM-generated content, yet LLMs often produce unreliable information (Maynez et al., 2020; Xu et al., 2023; Huang et al., 2023; Rawte et al., 2023).”
Quote in the paperSubstantial claim2 of 3 sources checked
- cites Maynez et al., 2020 source
  The Abstract reports hallucinations in neural-generated summaries, and §2 says summarization models need to be factual and faithful. Neither passage establishes that users rely on the accuracy of LLM-generated content; searches for user reliance, reader trust, and accuracy found no passage addressing that claim. Page numbers were not available in our copy. (LLM, conf=0.72)
  Experimental — accuracy not yet measured
- cites Xu et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Huang et al., 2023 source
Premisecitation
LLMs often produce unreliable information.
“Users rely on the accuracy of LLM-generated content, yet LLMs often produce unreliable information (Maynez et al., 2020; Xu et al., 2023; Huang et al., 2023; Rawte et al., 2023).”
Quote in the paperSubstantial claim2 of 3 sources checked
- cites Maynez et al., 2020 source
  The Abstract reports substantial hallucinated content in summaries generated by the neural summarization systems studied, and the Introduction gives examples of fabricated or factually wrong information. This supports the concern about unreliable generated content, but the paper studies abstractive summarization systems rather than LLMs generally. (LLM, conf=0.87)
  Experimental — accuracy not yet measured
- cites Xu et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Huang et al., 2023 source
Premisecitation
FACTSCORE extracts everything instead of just verifiable claims.
“Second, it extracts everything instead of just verifiable claims, an issue that is inherited by SAFE as in Figure 1.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Min et al., 2023 source
  Appendix A.2 (p. 15) says FACTSCORE splits each sentence into atomic facts, and Appendix A.5 (p. 16) acknowledges that an extracted atomic fact can be subjective. But the paper does not say it extracts everything indiscriminately: the Introduction (p. 1) says it evaluates biographies because they contain verifiable rather than subjective statements, and the Limitations (p. 9) restrict its applicability to less debatable facts. (LLM, conf=0.78)
  Experimental — accuracy not yet measured
Interpretationcitation
FACTSCORE and SAFE extract atomic claims with the implicit assumption that all claims are verifiable, leading to extraction of unverifiable claims.
“FACTSCORE and SAFE extract atomic claims with the implicit assumption that all claims are verifiable; unfortunately, this leads to the extraction of unverifiable claims (e.g., Claim 2 and 6 in Figure 1).”
Quote in the paperSubstantial claim2 of 2 sources checked
- cites Min et al., 2023 source
  Section 3.3 (p. 4) describes FActScore’s atomic-fact extraction, and Section 3.1 (p. 3) assumes that whether a fact is supported is undebatable. Appendix A.5 (p. 16) acknowledges that some extracted facts are subjective or debatable. The paper does not establish that this assumption causes unverifiable claims to be extracted, and it does not address SAFE. (LLM, conf=0.85)
  Experimental — accuracy not yet measured
- cites Wei et al., 2024 source
  The SAFE section and Appendix C.4 say SAFE splits responses into individual facts using a step almost identical to FActScore’s. Appendix C.10 gives an example of SAFE extracting “Maria is a refugee” from a hypothetical sentence, supporting the concern that extraction can produce a claim that cannot be checked as an actual event. The paper does not state that either method assumes all claims are verifiable or establish that such an assumption causes these extractions. (LLM, conf=0.84)
  Experimental — accuracy not yet measured
Premisecitation
Prior research targets error detection in individual sentences.
“Factual error detection: Prior research targets error detection in individual sentences (Mihaylova et al., 2019; Wadden et al., 2020; Shaar et al., 2022).”
Quote in the paperSubstantial claim1 of 3 sources checked
- cites Mihaylova et al., 2019
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Wadden et al., 2020 source
- cites Shaar et al., 2022
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Premisecitation
Forcing models to generate longer responses than necessary can elicit more infactual content.
“Forcing models to generate longer responses than necessary can also elicit more infactual content, as noticed by Min et al. (2023)”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Min et al., 2023 source
  Section 3.4 Results (p. 5) finds that facts appearing later in generated biographies have significantly higher error rates, supporting the concern about additional inaccurate content in longer responses. The paper does not test forcing models to respond longer than necessary, so it does not establish that intervention as the cause. (LLM, conf=0.84)
  Experimental — accuracy not yet measured
Findingown data
There are significant and intuitive differences in the ratio of verifiable claims to sentences across domains.
“We observe significant and intuitive differences in this ratio across domains”
Quote in the paperSubstantial claim
Findingown data
In the human-annotated claim verification data, 55% of claims were supported.
“Results in Table 2 show that only 55% of the claims are supported.”
Quote in the paperSubstantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Findingown data
VERISCORE's extraction and verification effectiveness is verified by human evaluations.
“we develop VERISCORE's extraction and verification in Section 3 by prompting closed LLMs, whose effectiveness is verified by human evaluations.”
Quote in the paperSubstantial claim
Findingown data
Splitting at conjunctions does not significantly shorten claims with inherently long core content.
“While some long claims could be split at conjunctions like and or or, this does not significantly shorten claims with inherently long core content, as seen in ( 5).”
Quote in the paperSubstantial claim
Findingown data
VERISCORE's extraction method correctly avoids extracting claims from non-factual content.
“Additionally, our method correctly avoids extracting claims from non-factual content.”
Quote in the paperSubstantial claim
Findingown data
Half of the 80 examined claims from GPT-4o generated ELI5 and FreshBooks responses were verified as unsupported.
“Half of these claims were verified as supported and the other half were not.17”
Quote in the paperNot a substantial claim
Findingown data
Human evaluation confirms open-weight VERISCORE performs competitively with closed VERISCORE.
“Human evaluation confirms open-weight VERISCORE performs competitively with closed VERISCORE”
Quote in the paperSubstantial claim
Findingcitation
Semantic or string matching fail to uncover possible hallucination.
“This indicates the need to move beyond semantic or string matching for verification as they fail to uncover possible hallucination.17”
Quote in the paperSubstantial claim
Findingown data
Mixt-8x22B-Inst-v0.1 is the best open LLM on VERISCORE.
“GPT-4o is the best closed LLM and Mixt-8x22B-Inst-v0.1 the best open LLM .”
Quote in the paperNot a substantial claim
Findingcitation
The content of supported claims is supported by search results via semantic or string match.
“The content is supported by search results via semantic or string match, as in ( 7).”
Quote in the paperSubstantial claim
Findingown data
Among the 41 fully agreed items, 15 are inconclusive.
“Among the 41 fully agreed items, 15 are inconclusive.”
Quote in the paperNot a substantial claim
Findingown data
GPT-4o is the best closed LLM on VERISCORE.
“GPT-4o is the best closed LLM and Mixt-8x22B-Inst-v0.1 the best open LLM .”
Quote in the paperNot a substantial claim
Findingown data
An LM's VERISCORE on one task does not necessarily correlate to its VERISCORE on a different task.
“We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., longform QA)”
- Also stated in: § 1 Introduction · plain
  “an LLM's factuality varies depending on the task and domain”
Quote in the paperSubstantial claim
Findingown data
Model size correlates with VERISCORE in long-form fact-seeking outputs.
“trend underscores the correlation of model size and VERISCORE in long-form fact-seeking outputs.”
Quote in the paperNot a substantial claim
Findingown data
SAFE often extracts trivial or vague claims that do not need to or cannot be verified.
“Third, SAFE often extracts trivial (4a) or vague claims (4b) that do not need to or cannot be verified.”
Quote in the paperNot a substantial claim
Findingown data
GPT-4o aligns best with human performance on the claim verification task among the tested LLMs.
“Results in Table 11 show that GPT-4o aligns the best with the human performance.”
Quote in the paperSubstantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Findingown data
For search results six to nine, their usefulness percentages range from 27.0% to 29.2%.
“For search results six to nine, their usefulness percentages range from 27.0% to 29.2%.”
Quote in the paperNot a substantial claim
Findingown data
The utility of the last search result is 13.3%.
“The utility of the last search result drops to only 13.3%.”
Quote in the paperNot a substantial claim
Findingown data
Human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks.
“human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks.”
Quote in the paperSubstantial claim
Findingown data
Gemma-2b-it performs the worst across all domains on VERISCORE.
“The smaller models fall behind on VERISCORE, with Gemma-2b-it performing the worse across all domains.”
Quote in the paperNot a substantial claim
Findingown data
Open-weight models such as Mixtral-8 × 22 are closing the gap with GPT-4o.
“open-weight models such as Mixtral-8 × 22 are closing the gap.”
Quote in the paperSubstantial claim
Findingown data
Closed models are more factual than open models on VERISCORE.
“Closed models are more factual.”
Quote in the paperSubstantial claim
Findingown data
Of the 50 triple-annotated items, 82% had complete agreement among the annotators.
“Of the 50 triple-annotated items, 82% had complete agreement among the annotators, and 14% had two annotators in consensus.”
Quote in the paperSubstantial claim
Findingown data
For claim extraction, the fine-tuned Mistral model trained on GPT-4 data achieves the most competitive performance.
“For claim extraction, the fine-tuned Mistral on GPT-4 data achieves the most competitive performance.”
Quote in the paperNot a substantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Findingreasoning
The unsupported claims do not have direct contradicting evidence.
“The unsupported claims do not have direct contradicting evidence.”
Quote in the paperSubstantial claim
Findingown data
GPT-4 was preferred more often than Claude 3 with our prompts.
“GPT-4 was preferred more often than Claude 3 with our prompts”
Quote in the paperSubstantial claim
Findingown data
No triple-annotated item is marked as inconclusive for the reason that there are both supporting and contradicting search results.
“Overall, no triple-annotated item is marked as inconclusive for the reason that there are both supporting and contradicting search results.”
Quote in the paperNot a substantial claim
Findingown data
Our approach is significantly favored even on biography generation.
“Notably, our approach is significantly favored even on biography generation.”
Quote in the paperNot a substantial claim
Findingown data
LMs' VERISCORE on two fact-seeking domains (e.g., ELI5 and Biography) do not necessarily correlate well.
“The Kendall's τ correlations between LMs' performance in domains in Figure 3 indicate that LMs' VERISCORE on two fact-seeking domains (e.g., ELI5 and Biography) do not necessarily correlate well.”
Quote in the paperSubstantial claim
Findingown data
Our method outperforms SAFE regardless of the model used.
“Our method outperforms SAFE regardless of the model used.”
Quote in the paperSubstantial claim
Findingown data
There are very few claims directly contradicted by the evidence list in the human study.
“In Section 3.3's human study, we notice that there are very few claims directly contradicted by the evidence list E c .”
Quote in the paperSubstantial claim
Findingown data
Half of the 80 examined claims from GPT-4o generated ELI5 and FreshBooks responses were verified as supported.
“Half of these claims were verified as supported and the other half were not.17”
Quote in the paperNot a substantial claim
Findingown data
14% of the 50 triple-annotated items had two annotators in consensus.
“Of the 50 triple-annotated items, 82% had complete agreement among the annotators, and 14% had two annotators in consensus.”
Quote in the paperNot a substantial claim
Findingown data
Overall, the GPT models perform better than the Claude 3 models on VERISCORE.
“Overall, the GPT models performs better than the Claude 3 models.15”
Quote in the paperNot a substantial claim
Findingown data
The fine-tuned Llama3 on GPT-4o data achieves F1 = 0.841 on the human annotated data in Section 3.3.
“The fine-tuned Llama3 on GPT-4o data performs the best on the human annotated data in Section 3.3, achieving F 1 = 0.841 (see Table 11).”
Quote in the paperNot a substantial claim
Findingown data
The smaller models fall behind on VERISCORE.
“The smaller models fall behind on VERISCORE, with Gemma-2b-it performing the worse across all domains.”
Quote in the paperNot a substantial claim
Findingown data
Mistral and GPT-4 extracted claims in Appendix F have an exact match rate of 43.7%.
“Mistral and GPT-4 extracted claims in Appendix F, the exact match rate is 43.7% and RougeL is 0.801.”
Quote in the paperNot a substantial claim
Findingown data
GPT-4o is the best-performing model overall across multiple long-form tasks.
“while GPT-4o is the best-performing model overall”
- Also stated in: § 1 Introduction · plain
  “GPT-4o generates the most factually-supported text when averaged across all datasets.”
Quote in the paperSubstantial claim
Findingown data
Fact-seeking domains (e.g., FreshBooks) have a higher density of verifiable claims.
“fact-seeking domains (e.g., FreshBooks) have a higher density of verifiable claims”
Quote in the paperSubstantial claim
Findingown data
This happens the most often in FreshBooks when a claim encapsulates aspects like someone's achievements or historical movements.
“This happens the most often in FreshBooks when a claim encapsulates aspects like someone's achievements or historical movements, as in (9-10).”
Quote in the paperNot a substantial claim
Findingcitation
Claims extracted from other fact-seeking generations are often long.
“However, claims extracted from other fact-seeking generations are often long.16”
Quote in the paperSubstantial claim
Findingown data
In a human study, VERISCORE's extraction method is preferred 93% of the time over SAFE's, even in biography generation.
“In a human study, our extraction method is preferred 93% of the time over SAFE's, even in biography generation.”
Quote in the paperSubstantial claim
Findingnone given
VERISCORE can be effectively implemented with either closed or fine-tuned open-weight language models.
“VERISCORE can be effectively implemented with either closed or fine-tuned open-weight language models”
Quote in the paperNot a substantial claim
Findingown data
DBRX-Instruct and the Mixtral models perform competitively to some versions of the GPT and Claude 3 models on VERISCORE.
“DBRX-Instruct and the Mixtral models performs competitively to some versions of the GPT and Claude 3 models.”
Quote in the paperNot a substantial claim
Findingown data
Top search results are more informative than lower-ranked search results.
“Top search results are more informative.”
Quote in the paperNot a substantial claim
Findingreasoning
Some unsupported claims require extensive supporting evidence.
“Some unsupported claims require extensive supporting evidence.”
Quote in the paperNot a substantial claim
Findingown data
Among 15 randomly picked inconclusive cases that were manually verified, two claims are not specified enough to be verified.
“Results show that two claims are not specified enough to be verified, for example, (11).”
Quote in the paperNot a substantial claim
Findingown data
The majority of Scruples data is invariably subjective and yields the 'No verifiable claim.' output.
“The reason of sampling less data points from Scruples is that the majority of them is invariably subjective and yields the”
Quote in the paperSubstantial claim
Findingown data
In cases where Mistral lost to GPT-4, Mistral misses small words and makes the extracted claims less specific.
“there are cases where Mistral lost to GPT-4 because it misses small words and makes the extracted claims less specific.”
Quote in the paperNot a substantial claim
Findingown data
In FreshQA, GPT-4o has a higher percentage of supported claims among all extracted claims (74.70%) compared to Claude 3 Haiku (72.71%).
“GPT-4o has a higher percentage of supported claim among all the extracted claims (74.70%) compared to Claude 3 Haiku (72.71%).”
Quote in the paperSubstantial claim
Findingown data
42.2% of the annotated items are labeled as inconclusive by the paper's annotators.
“As many as 42.2% of the annotated items are labeled as inconclusive by our annotators.”
Quote in the paperNot a substantial claim
Findingown data
Llama3-8B-Instruct fine-tuned for one epoch achieves the most balanced performance on supported and unsupported data points on the test and human data.
“The Llama3-8B-Instruct model fine-tuned for one epoch achieves the most balanced performance on supported and unsupported data points on the test and human data.”
Quote in the paperSubstantial claim
Findingown data
The majority of extracted claims from Gemma-2b-it and GPT-4o on WritingPrompts have the potential to be verified or falsified: 82.61% for Gemma and 71.15% for GPT-4o.
“It turns out that the majority of the claims (82.61% for Gemma and 71.15% for GPT-4o) has the potential to be verified/falsified.”
Quote in the paperSubstantial claim
Findingnone given
The open-weight implementation of VERISCORE's performance approaches that of the closed-weight implementation.
“We open-source both a closedand open-weight implementation of VERISCORE, with the latter's performance approaching that of the former.”
Quote in the paperNot a substantial claim
Findingown data
Human evaluations validate that VERISCORE is more effective than existing metrics for diverse long-form generation tasks that contain both verifiable and unverifiable content.
“Human evaluations validate that VERISCORE is more effective than existing metrics for diverse long-form generation tasks that contain both verifiable and unverifiable content.”
Quote in the paperSubstantial claim
Findingcitation
A human study in Section 3.1 confirms that SAFE falls short in less entity-centric domains.
“A human study in Section 3.1 confirms that SAFE falls short in less entity-centric domains.”
Quote in the paperSubstantial claim
Findingreasoning
The median number of verifiable claims (K) in FreshQA is four.
“The median number for verifiable claims (K) in FreshQA is four because the questions in general do not require long-form answers.”
Quote in the paperNot a substantial claim
Findingreasoning
For WritingPrompts, the VERISCORE of the models is very low.
“For the generations in WritingPromtps, the results in Table 16 show that the VERISCORE of the models is very low”
Quote in the paperNot a substantial claim
Findingown data
For FreshQA, Claude 3 models perform the best.
“For the generations in FreshQA, the results show that the Claude 3 models perform the best.”
Quote in the paperNot a substantial claim
Findingown data
The number of supported and unsupported items in the training dataset is imbalanced.
“Because the number of supported and unsupported items in the training dataset are imbalanced”
Quote in the paperSubstantial claim
Findingown data
Claims extracted by the paper's prompts are overwhelmingly preferred by annotators across all eight domains.
“The claims extracted by our prompts are overwhelmingly preferred by the annotators across all eight domains.”
Quote in the paperSubstantial claim
Findingown data
GPT-4o with ternary labels has the most balanced performance on supported and unsupported items among the tested models for claim verification.
“Overall, GPT-4o with ternary labels has the most balanced performance on the supported and unsupported items.”
Quote in the paperSubstantial claim
Findingown data
GPT-4 and fine-tuned Mistral perform similarly on claim extraction.
“Quality analysis GPT-4 and the fine-tuned Mistral perform similarly.”
Quote in the paperNot a substantial claim
Findingown data
GPT-3.5-turbo-1106 generates shorter responses than GPT-3.5-turbo-0613.
“GPT-3.5-turbo-1106 is an exception because the model generates shorter responses than GPT-3.5-turbo-0613,which hurts the recall.”
- Also stated in: § Prompt for Extracting Verifiable Claims from Question-Answering Type of Inputs · plain
  “On average, GPT-3.5-1106generates 8.56 sentences per response and GPT-3.5-0613generates 15.95.”
Quote in the paperSubstantial claim
Interpretationreasoning
FACTSCORE and SAFE are not suitable for most generation tasks because they assume that every claim is verifiable.
“These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false).”
Quote in the paperSubstantial claim
Interpretationreasoning
Such content might not be directly mentioned in search results but need to be inferred from a large body of documents.
“Such content might not be directly mentioned in search results but need to be inferred from a large body of documents.”
Quote in the paperNot a substantial claim
Interpretationown data
The verification task is feasible.
“The agreement result shows that the verification task is well-defined and feasible.”
Quote in the paperSubstantial claim
Interpretationreasoning
With the current system, it is hard to decide whether an unsupported claim is hallucinated because it is beyond what reasoning over search snippets can achieve.
“In sum, with the current system, it is hard to decide whether an unsupported claim is hallucinated because it is beyond what reasoning over search snippets can achieve.”
Quote in the paperNot a substantial claim
Interpretationreasoning
F1@K favors longer outputs.
“F 1@K favors longer outputs.”
Quote in the paperSubstantial claim
Interpretationown data
Our method effectively discriminates verifiable and unverifiable content.
“This level of variation shows that our method effectively discriminates verifiable and unverifiable content.”
Quote in the paperSubstantial claim
Interpretationown data
The verification task is well-defined.
“The agreement result shows that the verification task is well-defined and feasible.”
Quote in the paperSubstantial claim
Interpretationreasoning
Snippets do not offer enough background for such reasoning.
“Snippets do not offer enough background for such reasoning.”
Quote in the paperNot a substantial claim
Interpretationreasoning
Long and complex claims are likely to be marked as inconclusive.
“For these reasons, long and complex claims are likely to be marked as inconclusive.”
Quote in the paperSubstantial claim
Interpretationreasoning
Verifying the shorter claims does not mean the longer one is verified.
“However, verifying the shorter claims does not mean the longer one is verified because of solidified.”
Quote in the paperSubstantial claim
Interpretationreasoning
Comprehensive longform factuality evaluation requires multiple tasks, not just biography generation.
“multiple tasks (not just biography generation) are needed for comprehensive longform factuality evaluation because an LLM's factuality varies depending on the task and domain”
Quote in the paperSubstantial claim
Interpretationnone given
VERISCORE is the first approach that considers inter-sentence context when extracting claims.
“Second, VERISCORE is the first approach that considers inter-sentence context when extracting claims, removing the need for expensive claim revision steps present in SAFE”
Quote in the paperNot a substantial claim
Interpretationown data
Open-domain claim verification requires extensive reasoning to verify the connection between parts of a claim, beyond identifying exact or related terms.
“the low supported rate showcases that open-domain claim verification is beyond identifying exact or related terms but requires extensive reasoning to verify the connection between parts of a claim.”
Quote in the paperSubstantial claim
- Rests on an assumption: The claims in the human-annotated verification data are verifiable in principle.
Interpretationreasoning
LMs exhibit varying strengths in different domains.
“This suggests that LMs exhibit varying strengths in different domains, highlighting the need for diverse tasks to comprehensively assess LMs' factuality.”
Quote in the paperSubstantial claim
Interpretationreasoning
For domains that do not require long generations, models that generate short to-the-point outputs will be penalized if other models generate lengthy outputs with auxiliary information.
“However, for domains that do not require long generations, models that generate short to-the-point outputs will be penalized if other models generate lengthy outputs with auxiliary information (e.g., FreshQA in Appendix J).”
Quote in the paperSubstantial claim
Premisecitation
Li et al. (2023) test LLMs' detection of synthesized factual errors at the paragraph level.
“At the paragraph level, Li et al. (2023) test LLMs' detection of synthesized factual errors but do not locate the errors.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Li et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Interpretationreasoning
The shorter responses generated by GPT-3.5-turbo-1106 hurt recall.
“GPT-3.5-turbo-1106 is an exception because the model generates shorter responses than GPT-3.5-turbo-0613,which hurts the recall.”
Quote in the paperNot a substantial claim
Interpretationown data
SAFE's relevance check step negatively impacts evaluation.
“Third, the relevance check step negatively impacts evaluation.”
Quote in the paperSubstantial claim
Premisecitation
FEVEROUS and AVERITEC remained limited to sentence-level facts.
“FEVEROUS (Aly et al., 2021) and AVERITEC (Schlichtkrull et al., 2023) build on FEVER but remained limited to sentence-level facts.”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites Aly et al., 2021
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Schlichtkrull et al., 2023 source
Premisecitation
Li et al. (2023) do not locate the errors.
“At the paragraph level, Li et al. (2023) test LLMs' detection of synthesized factual errors but do not locate the errors.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Li et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims. 36
Classified as probably not a substantial claim. 76
The quote only partly states this claim. 4

May combine two claims.
Interpretationreasoning
FACTSCORE's extraction method is optimized for biographies and is inapplicable to other domains.
“However, their extraction method is optimized for biographies and is inapplicable to other domains.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Min et al., 2023 source
  The Limitations section (p. 9) says the experiments focus on biographies but explicitly states that FACTSCORE can be applied to broader domains. Appendix B.5 (p. 19) reports a small-scale application to NLP papers and says this suggests generalization beyond biographies. The paper therefore contradicts the claim that the method is inapplicable to other domains, while leaving broader validation for future work. (LLM, conf=0.91)
  Checked by a reasoning model — may be wrong.
  Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
Premisecitation
The F1@K metric from SAFE considers both factual precision and recall.
“We adopt the F 1 @K metric from SAFE, which considers both factual precision and recall.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Wei et al., 2024 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Later facts in long responses tend to be less accurate.
“They provide more details but, as Min et al. (2023) shows, later facts in long responses tend to be less accurate.”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Classified as probably not a substantial claim.
Premisecitation
F1@K considers both factual precision and recall.
“F 1 @K (Wei et al., 2024) considers both factual precision and recall, which improves on measuring factual precision alone (Min et al., 2023).”
Quote in the paperSubstantial claim2 of 2 sources checked
- cites Wei et al., 2024 source
- cites Min et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
The overall evaluation pipeline improves models' trustworthiness.
“The overall evaluation pipeline helps generate post hoc citations and iteratively improve model generations' factuality, which improves models' trustworthiness model (Huang and Chang, 2024; Ye et al., 2024).”
Quote in the paperNot a substantial claim1 of 2 sources checked
- cites Huang and Chang, 2024 source
  Section 4.2.1 describes post-hoc evaluation and says combining it with pre-hoc retrieval could produce more accurate, well-supported content; the Introduction says citations could bolster credibility. The paper presents these as potential benefits, not as evidence that an overall evaluation pipeline improves model trustworthiness, and it does not describe iterative improvement of generations’ factuality. (LLM, conf=0.85)
  Experimental — accuracy not yet measured
- cites Ye et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation
Research has focused on enhancing factual precision and identifying inaccuracies.
“Research has thus focused on enhancing factual precision (Lin et al., 2024) and identifying inaccuracies.”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Lin et al., 2024 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
FEVER features synthesized incorrect sentences from Wikipedia.
“FEVER (Thorne et al., 2018) features synthesized incorrect sentences from Wikipedia.”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Thorne et al., 2018 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Retrieval is commonly used in factual error detection.
“It is commonly used in factual error detection (Min et al., 2023; Wei et al., 2024; Thorne et al., 2018) and helps evaluation by providing up-to-date knowledge.”
Quote in the paperNot a substantial claim3 of 3 sources checked
- cites Min et al., 2023 source
- cites Wei et al., 2024 source
- cites Thorne et al., 2018 source
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationcitation
VERISCORE's definition of verifiable claims advances beyond the concept of atomic facts but remains a working definition rather than a formal one.
“Although our definition advances beyond the concept of atomic facts (Min et al., 2023), it remains a working definition rather than a formal one.”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
World knowledge can be achieved by employing retrieval.
“Factuality evaluation often requires world knowledge, which can be achieved by employing retrieval (Ram et al., 2023; Vu et al., 2023).”
Quote in the paperNot a substantial claim1 of 2 sources checked
- cites Ram et al., 2023 source
- cites Vu et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Decomposing a piece of long-form text into shorter sentences or search queries for factuality evaluation is commonly implemented in previous works.
“Decomposing a piece of long-form text into shorter sentences or search queries for factuality evaluation is commonly implemented in previous works (Kamoi et al., 2023; Gao et al., 2023; Wang et al., 2023; Min et al., 2023; Chern et al., 2023; Wanner et al., 2024; Wei et al., 2024; Guan et al., 2024; Chen et al., 2024).”
Quote in the paperNot a substantial claim2 of 3 sources checked
- cites Kamoi et al., 2023 source
- cites Gao et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Wang et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Factuality evaluation often requires world knowledge.
“Factuality evaluation often requires world knowledge, which can be achieved by employing retrieval (Ram et al., 2023; Vu et al., 2023).”
Quote in the paperNot a substantial claim1 of 2 sources checked
- cites Ram et al., 2023 source
- cites Vu et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
The decomposition helps locate factual errors.
“The decomposition helps locate factual errors and offers a fine-grained estimate of models' factuality (Min et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
The decomposition offers a fine-grained estimate of models' factuality.
“The decomposition helps locate factual errors and offers a fine-grained estimate of models' factuality (Min et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
The overall evaluation pipeline helps generate post hoc citations.
“The overall evaluation pipeline helps generate post hoc citations and iteratively improve model generations' factuality, which improves models' trustworthiness model (Huang and Chang, 2024; Ye et al., 2024).”
Quote in the paperNot a substantial claim1 of 2 sources checked
- cites Huang and Chang, 2024 source
- cites Ye et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Retrieval helps evaluation by providing up-to-date knowledge.
“It is commonly used in factual error detection (Min et al., 2023; Wei et al., 2024; Thorne et al., 2018) and helps evaluation by providing up-to-date knowledge.”
Quote in the paperNot a substantial claim3 of 3 sources checked
- cites Min et al., 2023 source
  Section 4.2 (p. 7) reports that retrieval improves factual evaluation by providing context the evaluator may not have memorized. It does not say that the retrieved knowledge is up to date, so that qualifier is unsupported. (LLM, conf=0.86)
  Experimental — accuracy not yet measured
- cites Wei et al., 2024 source
  The paper’s “Rating individual facts” section says SAFE uses Google Search results to verify facts, and Appendix C.7 reports that open-web search can find information unavailable on Wikipedia. This supports retrieval as an aid to factuality evaluation, but the passages do not establish that its benefit comes specifically from providing up-to-date knowledge. (LLM, conf=0.82)
  Experimental — accuracy not yet measured
- cites Thorne et al., 2018 source
  Sections 5.2 (Evaluation) and 5.3 (Document Retrieval) describe retrieving evidence to verify claims and evaluate whether a system returns the correct evidence. They do not say retrieval provides up-to-date knowledge; Section 3.1 describes using a June 2017 Wikipedia dump. Page numbers are unavailable in our copy. (LLM, conf=0.85)
  Experimental — accuracy not yet measured
Classified as probably not a substantial claim.
Premisecitation
Later content in model responses tends to be less factual.
“as noticed by Min et al. (2023) that later content in model responses tends to be less factual.”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Min et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
In the human-annotated claim verification data, 40% of claims were labeled inconclusive (a).
“Inconclusive (a) 128 40%”
Quote in the paperNot a substantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Classified as probably not a substantial claim.
Findingnone given
Search results may be insufficient for verification because expertise or extensive logical reasoning is often needed.
“Two issues are identified: (1) not all claims can be short, and long claims are harder to verify, and (2) search results may be insufficient as expertise or extensive logical reasoning is often needed for verification.”
Quote in the paperSubstantial claim
The quote only partly states this claim.
Findingown data
The LMs' performance was worse with a ternary classification prompt than with the binary classification prompt.
“We also experimented with a ternary classification prompt but the LMs' performance was worse.”
Quote in the paperNot a substantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
In the human-annotated claim verification data, 2.8% of claims were contradicted.
“Claim contradicted 9 2.8%”
Quote in the paperNot a substantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
Another reason a claim is inconclusive is that there is no direct mentioning of a part of the claim or no evidence verifies the connection between the parts of a claim.
“Second, there is no direct mentioning of a part of the claim or no evidence verifies the connection between the parts of a claim (see Table 8).”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
WritingPrompts' creative story outputs contain almost no verifiable claims with a ratio of 0.03, despite containing the longest responses.
“WritingPrompts' creative story outputs contain almost no verifiable claims with a ratio of 0.03, despite containing the longest responses.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
There is a gap between closed and open-weight LMs, with GPT-4o achieving significantly higher VERISCORE than any open LM.
“Our results highlight the gap between closed and open-weight LMs, which GPT-4o achieving significantly higher VERISCORE than any open LM.”
Quote in the paperSubstantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
In the human-annotated claim verification data, 2.2% of claims were labeled inconclusive (b).
“Inconclusive (b) 7 2.2%”
Quote in the paperNot a substantial claim
- Rests on an assumption: The human annotations of claim support are correct.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Occasionally, shorter claims can be extracted from a longer one.
“Occasionally, shorter claims can be extracted from a longer one, as the bracketed content in (6).”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
SAFE indiscriminately extracts everything, such as subjective content and personal experience.
“First, it indiscriminately extracts everything, such as subjective content (2a) and personal experience (2b).”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
VERISCORE's extraction method effectively addresses the issue of unresolved referents and eliminates the need of claim revision and removal.
“It effectively addresses the issue of unresolved referents and eliminates the need of claim revision and removal.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Only over half of the claims are supported.
“Only over half of the claims are supported.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
The Fleiss κ is 0.7316, indicating substantial agreement.
“The Fleiss κ is 0.7316 (substantial agreement).”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Findingnone given
Not all claims can be short.
“Two issues are identified: (1) not all claims can be short, and long claims are harder to verify, and (2) search results may be insufficient as expertise or extensive logical reasoning is often needed for verification.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Findingreasoning
Unsupported claims are unsupported because there is no direct mention of (parts of) the claims or the connection between the parts of the claims.
“They are unsupported because there is no direct mention of (parts of) the claims or the connection between the parts of the claims.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Some long claims can be split at conjunctions like and or or.
“While some long claims could be split at conjunctions like and or or, this does not significantly shorten claims with inherently long core content, as seen in ( 5).”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Findingown data
The supported claims resemble encyclopedic writing.
“The supported claims resemble encyclopedic writing.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S2 (checkable).
Findingnone given
Long claims are harder to verify.
“Two issues are identified: (1) not all claims can be short, and long claims are harder to verify, and (2) search results may be insufficient as expertise or extensive logical reasoning is often needed for verification.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
SAFE overly decomposes texts, causing meaning overlaps between claims, which can disproportionately affect the final score.
“Second, SAFE overly decomposes texts, causing meaning overlaps between claims as in (3), which can disproportionately affect the final score.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Example (8) is judged as unsupported because there is no mention of the Meiji era and stuffed tigers occurring together in the search results.
“Example ( 8) is judged as unsupported because there is no mention of the Meiji era and stuffed tigers occurring together in the search results.19”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Evaluating 400 GPT-4o generations in the domains in Table 16 using the prompting method cost $1,038 USD.
“Evaluating 400 GPT-4o generations in the domains in Table 16 using our prompting method cost $1,038 USD.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The annotators identified three major issues with SAFE's extraction pipeline.
“The annotators identified three major issues with SAFE's extraction pipeline.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S4 (based).
Findingnone given
Verifying complex, lengthy assertions (common in many long-form tasks such as LFQA) against Google Search results can fail due to challenges in retrieving relevant documents from such queries.
“verifying complex, lengthy assertions (common in many long-form tasks such as LFQA) against Google Search results can fail due to challenges in retrieving relevant documents from such queries.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
One reason a claim is inconclusive is that it is too general to be verified.
“First, a claim is too general to be verified”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Of the 360 annotated items, claims extracted by SAFE were preferred only 26 times, with 19 of those preferences being marginal.
“Of the 360 annotated items, claims extracted by SAFE were preferred only 26 times, with 19 of those preferences being marginal.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Mistral and GPT-4 extracted claims in Appendix F have a RougeL of 0.801.
“Mistral and GPT-4 extracted claims in Appendix F, the exact match rate is 43.7% and RougeL is 0.801.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
On average, FACTSCORE's claims have 7 words, with the longest one having 18 words.
“On average, FACTSCORE's claims have 7 words, with the longest one having 18 words.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
In the experiments, each response on average has 40 sentences, or 20 sentences on average if excluding WritingPrompts responses.
“In our experiments, each response on average has 40 sentences (20 sentences on average if excluding WritingPrompts responses).”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Findingnone given
Complex claims (e.g., not entity-centric or formulaic) are challenging to verify against search results.
“Finally, we notice that complex claims (e.g., not entity-centric or formulaic) are challenging to verify against search results.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Annotation disagreements can arise when a part of a claim cannot be verified by the search results.
“-a part of a claim cannot be verified by the search results”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
On average, ELI5 claims have 12 words, with the longest one having 25 words.
“In contrast, the ELI5 claims on average have 12 words, with the longest one having 25 words.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Short marked sentences are usually the numbering of numbered lists.
“These short marked sentences are usually the numbering of numbered lists.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Applying SAFE's extraction pipeline to five texts, 11% of 211 claims were removed, of which 58% were actually relevant.
“we applied SAFE's extraction pipeline to five texts and examined which claims were removed.”
- Also stated in: § A Weaknesses in FACTSCORE and SAFE · plain
  “It turns out that 11% of 211 claims were removed, of which 58% were actually relevant.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
In total, the fine-tuning data contains 99592 input-output pairs, among which 902 pairs have a short marked sentence, and 31819 pairs have 'No verifiable claim.' as the output.
“In total, we got 99592 input-output pairs, among which 902 pairs have a short marked sentence, and 31819 pairs have”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.
Findingown data
Mistral misses small words, and as a result, the claims it extracts are less specific than those extracted by GPT-4.
“Mistral misses small words, hence the extracted claims are less specific”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
There are 9 items in the human study on claim verification on which annotators did not reach full agreement.
“There are 9 items in the human study in Section 3.3 on which the annotators did not reach a full agreement.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
SAFE's prompt templates of claim revision and relevance check alone, without filling in content, cost about $1.7 per 100 claims.
“SAFE's prompt templates of claim revision and relevance check alone, without filling in content, cost about $1.7 per 100 claims (estimated using https:// platform.openai.com/tokenizer).”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The string-based metric Rouge1 score is 0.8243.
“The string-based metrics and the scores are the following: exact match = 0.4317, Rouge1 = 0.8243, Rouge2 = 0.7576, RougeL = 0.8009, and CHRF++ = 74.6686.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The string-based metric Rouge2 score is 0.7576.
“The string-based metrics and the scores are the following: exact match = 0.4317, Rouge1 = 0.8243, Rouge2 = 0.7576, RougeL = 0.8009, and CHRF++ = 74.6686.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.
Findingreasoning
Two annotators chose the supported label for the claim about traits that can be chosen, while one chose inconclusive; the annotator's comment states that the claim is 'can be chosen' and all results are that they potentially 'could' be chosen, not that they actually CAN.
“Two annotators chosen the supported label but one chose inconclusive.The annotator's comment states:”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Both FreshQA and WritingPrompts domains yield very few verifiable facts.
“As shown in Table 16, both domains yield very few verifiable facts.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The string-based metric RougeL score is 0.8009.
“The string-based metrics and the scores are the following: exact match = 0.4317, Rouge1 = 0.8243, Rouge2 = 0.7576, RougeL = 0.8009, and CHRF++ = 74.6686.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingreasoning
Two annotators chose the inconclusive label for the claim about Luis Guillermo Rivera, while one chose supported; this is an annotator mistake because the name Luis Guillermo Rivera is not mentioned in the evidence.
“Two annotators chose the inconclusive label for this claim but one chose supported based on one search result as given below.This is an annotator mistake because the name Luis Guillermo Rivera is not mentioned in the evidence.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
On average, Claude 3 Haiku has 5.5 claims per response, higher than K = 4, but GPT-4o has only 3.39.
“On average, Claude 3 Haiku has 5.5 claims per response, higher than K = 4, but GPT-4o has only 3.39.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
There are four sources of disagreement in the human verification study: annotator mistake, disagreement in interpretation, claim complexity, and indirect evidence requiring intermediate reasoning.
“After inspecting these items, we conclude 4 sources of disagreement, listed in Table 9 with examples.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
VERISCORE takes about two hours to verify 10k claims.
“For verification, it takes about two hours to verify 10k claims.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The string-based metric exact match score is 0.4317.
“The string-based metrics and the scores are the following: exact match = 0.4317, Rouge1 = 0.8243, Rouge2 = 0.7576, RougeL = 0.8009, and CHRF++ = 74.6686.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
From the generated verification data, 13403 data points were randomly sampled, among which 9996 have the supported label and 3407 have the unsupported label.
“From the generated data, we randomly sampled 13403 data points, among which 9996 has the supported label and 3407 has the unsupported label.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Findingnone given
Mistral puts multiple pieces of information into one claim, while GPT-4 breaks the information down into multiple claims.
“Mistral puts multiple pieces of information into one claim while GPT-4 breaks the information down into multiple claims”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
The remaining 42% of removed claims were either tautologies or not claims and should not have been extracted.
“The remaining 42% were either tautologies or not claims and should not have been extracted.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
In certain cases, Mistral does not refrain itself to the marked sentence in an input but also extracts claims that come after the span.
“in certain cases, Mistral does not refrain itself to the marked sentence in an input but also extracts claims that come after the span.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The string-based metric CHRF++ score is 74.6686.
“The string-based metrics and the scores are the following: exact match = 0.4317, Rouge1 = 0.8243, Rouge2 = 0.7576, RougeL = 0.8009, and CHRF++ = 74.6686.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingcitation
The two authors fully agreed in 106 data points, achieving Cohen's κ = 0.4320, indicating moderate agreement.
“The two authors fully agreed in 106 data points, achieving a Cohen's κ = 0.4320 (moderate agreement, Landis and Koch, 1977).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Landis and Koch, 1977 source
May combine two claims.
Findingown data
GPT-4o tends to generate short and to-the-point answers while Claude 3 Haiku tends to generate longer answers, offering more auxiliary information.
“GPT-4o tends to generate short and to-the-point answers while Claude 3 Haiku tends to generate longer answers, offering more auxiliary information.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Processing 100 claims without parallelization takes 35 minutes using GPT-4.
“Processing 100 claims without parallelization takes 35 minutes using GPT-4.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
For model responses that are extremely infactual, VERISCORE's claim extractor might still extract a small amount of unverifiable claims.
“for model responses that are extremely infactual (e.g., WritingPrompts in Appendix J), our claim extractor might still extract a small amount of unverifiable claims.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning
With one RTX8000 GPU, VERISCORE takes about 4 hours to extract claims from 400 GPT-4o responses without parallelization.
“With one RTX8000 GPU, it takes about 4 hours to extract claims from 400 GPT-4o responses without parallelization.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Annotation disagreements can arise when the entity or person mentioned in the claim has no clear referent.
“-the entity/person mentioned in the claim has no clear referent (e.g.,”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
Occasionally, Mistral puts multiple pieces of information into one claim while GPT-4 breaks the information down into multiple claims.
“Occasionally, Mistral puts multiple pieces of information into one claim while GPT-4 breaks the information down into multiple claims.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Among 360 annotated data points, claims extracted by SAFE are only preferred 26 times by the three annotators in total, among which 19 were chosen hesitatingly.
“Among the 360 annotated data points, the claims extracted by SAFE are only preferred 26 times by the three annotators in total, among which 19 were chosen hesitatingly, as indicated by the light red color in Figure 5.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
On average, GPT-3.5-1106 generates 8.56 sentences per response and GPT-3.5-0613 generates 15.95 sentences per response.
“On average, GPT-3.5-1106generates 8.56 sentences per response and GPT-3.5-0613generates 15.95.”
Quote in the paperSubstantial claim
May combine two claims.
Findingreasoning
WritingPrompts generations contain very few verifiable claims, resulting in K = 1.
“Hence, the generations contain very few verifiable claims, resulting in K = 1.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given
Annotation disagreements can arise when a part of a claim is supported and contradicted by different pieces of evidence.
“-a part of a claim is supported and contradicted by different pieces of evidence”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given
FreshQA mostly requires short answers.
“the former mostly requires short answers”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationown data
Achieving atomicity is hard, as exemplified by Example (1).
“Achieving atomicity is also hard as exemplified by Example (1).”
Quote in the paperNot a substantial claim
Classified as probably not a substantial claim.
Study design and conductnone given
To verify our extraction method's efficacy, we conducted a pairwise comparison of claims extracted by VERISCORE and SAFE with three human raters.
“To verify our extraction method's efficacy, we conducted a pairwise comparison of claims extracted by VERISCORE and SAFE with three human raters.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given
WritingPrompts is not fact-seeking.
“the latter is not fact-seeking”
Quote in the paperNot a substantial claim
May combine two claims.
Study design and conductown data
The three annotators fully agreed on 99 out of 120 annotated data points, resulting in a Fleiss κ = 0.7662 (substantial agreement).
“The three annotators fully agreed on 99 out of 120 annotated data points, resulting in a Fleiss κ = 0.7662 (substantial agreement, Landis and Koch, 1977).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Landis and Koch, 1977 source
Classified as probably not a substantial claim.
Premisecitation
Shorter, self-contained claims help locate factual errors.
“Shorter, self-contained claims are desired because they help locate factual errors and are easy to be verified as employed by FACTSCORE for biography.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites FACTSCORE
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Interpretationnone given
VERISCORE is a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content.
“We address this issue with VERISCORE, 1 a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Study design and conductreasoning
For time and cost efficiency, we only used SAFE's fact extraction and revision steps.
“For time and cost efficiency, we only used SAFE's fact extraction and revision steps (see Appendix A and Footnote 3).”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Premisecitation
Shorter, self-contained claims are easy to verify.
“Shorter, self-contained claims are desired because they help locate factual errors and are easy to be verified as employed by FACTSCORE for biography.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites FACTSCORE
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Study design and conductreasoning
To ensure the comparison was independent of the model used, we used GPT-4, paired with SAFE's and our methods, to extract claims from half of the data points and Claude 3 for the other half.
“To ensure the comparison was independent of the model used, we used GPT-4, paired with SAFE's and our methods, to extract claims from half of the data points and Claude 3 for the other half.”
Quote in the paperSubstantial claim
May combine two claims. The quote only partly states this claim.
Interpretationown data
The fine-tuned VERISCORE pipeline saves considerable money compared to the prompting method, making evaluation more accessible.
“The fine-tuned VERISCORE saves considerable money, making the evaluation process more accessible.”
Quote in the paperSubstantial claim
May combine two claims. The quote only partly states this claim.
Interpretationreasoning
SAFE's extraction pipeline is multistep and adds significant processing time and cost because it must resolve vague references via claim revision and relevance check.
“SAFE's extraction pipleine is multistep.”
- Also stated in: § A Weaknesses in FACTSCORE and SAFE · plain
  “The entire pipeline adds significant processing time and cost.”
Quote in the paperNot a substantial claim
May combine two claims.
Interpretationown data
GPT-4o generates shorter outputs than Claude 3 Haiku, contributing to GPT-4o's lower final VERISCORE.
“The fact that GPT-4o generates shorter outputs than Claude 3 Haiku contributes to the lower final VERISCORE of GPT-4o.”
Quote in the paperSubstantial claim
May combine two claims.
Interpretationown data
Within the Claude 3 family, Claude 3 Haiku generates longer outputs than the other two and has more responses with 4 or more supported claims, resulting in a higher VERISCORE.
“Within the Claude 3 model family, Claude 3 Haiku generates longer outputs than the other two and has more responses with 4 or more supported claims, resulting in a higher VERISCORE.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
The overall evaluation pipeline iteratively improves model generations' factuality.
“The overall evaluation pipeline helps generate post hoc citations and iteratively improve model generations' factuality, which improves models' trustworthiness model (Huang and Chang, 2024; Ye et al., 2024).”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites Huang and Chang, 2024 source
  Checked before — its earlier verdict is held back until a new check runs.
- cites Ye et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable).
Premisecitation
Detecting factual errors in a long-form text at once is hard.
“Long-form factuality evaluation: Detecting factual errors in a long-form text at once is hard (Li et al., 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Li et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Premisecitation
FEVEROUS and AVERITEC build on FEVER.
“FEVEROUS (Aly et al., 2021) and AVERITEC (Schlichtkrull et al., 2023) build on FEVER but remained limited to sentence-level facts.”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites Aly et al., 2021
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Schlichtkrull et al., 2023 source
May combine two claims.
Premisereasoning
SAFE's performance is only evaluated on FACTSCORE's biography data despite being applied to 38 fact-seeking topics.
“Despite being applied to 38 factseeking topics, SAFE's performance is only evaluated on FACTSCORE's biography data.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown data
SAFE has a propensity to extract unverifiable claims.
“We also show SAFE's extracted claims from the same text to highlight its propensity to extract unverifiable claims (Claim 2 and 6)”
Quote in the paperNot a substantial claim
May combine two claims.
Interpretationown data
The VERISCORE pipeline works effectively on fictional content although it occasionally extracts unverifiable claims.
“The results of the models on WritingPrompts prove that the VERISCORE pipeline works effectively on fictional content although it occasionally extracts unverifiable claims.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Interpretationreasoning
Adding the question or the first sentence of a paragraph into context1 can help a model better recover time, location, and person references in a claim.
“Adding the question or the first sentence of a paragraph into context1 can help a model better recover time, location, and person references in a claim.”
Quote in the paperSubstantial claim

101 of 101

The paper relies on this without support.
The claims in the human-annotated verification data are verifiable in principle.
“the low supported rate showcases that open-domain claim verification is beyond identifying exact or related terms but requires extensive reasoning to verify the connection between parts of a claim.”
The paper relies on this without support.
The human annotations of claim support are correct.
“Results in Table 2 show that only 55% of the claims are supported.”
The paper relies on this without support.
Google search results provide sufficient evidence to verify the factuality of claims.
“You need to judge whether a claim is supported or contradicted by Google search results, or whether there is no enough information to make the judgement (i.e., inconclusive).”

3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important.
“Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important.”
From All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations (2025) · cites this paper as Song et al., 2024
Section 2.4 (page unavailable) says VERISCORE adopts SAFE’s F1@K metric and calculates it from counts of supported claims, without assigning different importance weights to those claims. This supports the point for that metric, but the paper does not establish that all existing factuality-evaluation methods treat every claim as equally important; Section 2.1.1 also describes SAFE’s relevance check, which decides whether a claim is worth checking. (LLM, conf=0.78)
Experimental — accuracy not yet measured
The authors evaluate factuality with VeriScore F1@k (median over prompts), which decomposes each generated answer into claims and uses a verifier to label each claim as supported, unsupported, or insufficient given evidence snippets.
“We evaluate factuality with VeriScore (Song et al., 2024) F1@k (median over prompts), which decomposes each generated answer into claims and uses a verifier to label each claim as supported, unsupported, or insufficient given evidence snippets (Wei et al., 2024); see Appendix E.”
From MIRAGE: Defending Long-Form RAG Against Misinformation Pollution (2026) · cites this paper as Song et al., 2024
Sections 2.2–2.4 describe extracting claims, checking them against retrieved evidence, and calculating F1@K. But §2.4 defines VeriScore as the average F1@K across prompt responses; the median sets K from the number of extracted claims, not the score over prompts. Section 2.3 says the final verifier combines contradicted and inconclusive cases into one unsupported category, rather than using the three labels stated in the claim. Page numbers were not available in our copy. (LLM, conf=0.91)
Experimental — accuracy not yet measured
Existing factuality systems are a poor match for the medical domain because they are typically evaluated on objective, entity-centric, formulaic texts such as biographies and historical topics.
“However, existing factuality systems are a poor match for the medical domain, as they are typically only evaluated on objective, entity-centric, formulaic texts such as biographies and historical topics.”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Song et al., 2024
The Introduction says FACTSCORE and SAFE are limited by assumptions suited to fact-dense, formulaic text such as biographies. Section 2.1.1 says FACTSCORE’s extraction method was optimized for biographies and SAFE’s performance was evaluated only on biography data. The paper does not establish that existing systems are typically evaluated on historical topics or specifically assess their suitability for the medical domain. (LLM, conf=0.84)
Experimental — accuracy not yet measured
VeriScore fine-tuned a model for the decomposition step, largely solving the hallucinated, incorrectly structured, and redundant issues.
“Also, they fine-tuned a model for the decomposition step, largely solving the hallucinated, incorrectly structured, and redundant issues as well.”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Song et al., 2024
Section 4.1 and Appendix F report fine-tuning an open-weight model for claim extraction, VeriScore’s decomposition step. But the Manual quality comparison says the fine-tuned model still sometimes combines information that GPT-4 separates and extracts claims outside the marked sentence. The paper does not establish that fine-tuning largely solved hallucinated, incorrectly structured, and redundant outputs. (LLM, conf=0.87)
Experimental — accuracy not yet measured
VeriScore expands the generation domains to include fictional stories, Wikipedia, biographies, and Reddit.
“VeriScore (Song et al., 2024) expands the generation domains to include fictional stories, Wikipedia, biographies, and an online forum, Reddit.”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Song et al., 2024
Table 1 (section 3.2; page unavailable) includes WritingPrompts stories, biographies, Reddit-sourced datasets, and Wikipedia articles. But it marks the Wikipedia dataset, wikitext-103, for human evaluation of claim extraction only, not as a generation domain; Appendix J explicitly discusses fictional WritingPrompts generations. (LLM, conf=0.86)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Annotators' hourly pay differs between sections
Minor
The ethics statement gives an hourly rate of $18, while Appendix D says each annotator was compensated at $15 per hour. The ethics statement describes bonuses as additional, so they do not reconcile the stated base rates.

1 of 1

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 14

Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop on Fact Extraction and VERifica- tion (FEVER), pages 1-13, Dominican Republic. Association for Computational Linguistics. · DOI
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-world entailment for claims in Wikipedia. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7561-7583, Singapore. As- sociation for Computational Linguistics. · DOI · Paper page
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hal- lucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449-6464, Singapore. Association for Com- putational Linguistics. · DOI · Paper page
Claudia Maienborn. 2003. Event-internal modifiers: Semantic underspecification and conceptual interpre- tation, pages 475-510. De Gruyter Mouton, Berlin, Boston. · DOI
Claudia Maienborn. 2019. Events and States. In The Oxford Handbook of Event Structure. Oxford Univer- sity Press. · DOI
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics. · DOI · Paper page
Tsvetomila Mihaylova, Georgi Karadzhov, Pepa Atanasova, Ramy Baly, Mitra Mohtarami, and Preslav Nakov. 2019. SemEval-2019 task 8: Fact checking in community question answering forums. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 860-869, Minneapo- lis, Minnesota, USA. Association for Computational Linguistics. · DOI
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singa- pore. Association for Computational Linguistics. · DOI · Paper page
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented lan- guage models. Transactions of the Association for Computational Linguistics, 11:1316-1331. · DOI · Paper page
Shaden Shaar, Firoj Alam, Giovanni Da San Martino, and Preslav Nakov. 2022. The role of context in detecting previously fact-checked claims. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1619-1631, Seattle, United States. Association for Computational Linguistics. · DOI
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics. · DOI · Paper page
Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3225-3245, Toronto, Canada. Association for Com- putational Linguistics. · DOI · Paper page

Sorted by citations

HalluLens: LLM Hallucination Benchmark
Bang, Y., Ji, Z., Schelten, A., Hartshorn, A., Fowler, T., Zhang, C., Cancedda, N., & Fung, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1176 · Cited by 41 · see citing works
Verifying Facts in Patient Care Documents Generated by Large Language Models Using Electronic Health Records
Chung, P., Swaminathan, A., Goodell, A. J., Kim, Y., Reincke, S. M., Han, L., Deverett, B., Sadeghi, M. A., Ariss, A.-B., Ghanem, M., et al. (2025) · NEJM AI · doi:10.1056/aidbp2500418 · Cited by 37 · see citing works
Self-citation
Explanatory Summarization with Discourse-Driven Planning
Liu, D., Yu, X., Demberg, V., & Lapata, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.30 · Cited by 2 · see citing works
Can LLMs Automate Fact-Checking Article Writing?
Sahnan, D., Corney, D. P. A., Larraz, I., Zagni, G., Míguez, R., Xie, Z., Gurevych, I., Churchill, E. F., Chakraborty, T., & Nakov, P. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.644 · Cited by 2 · see citing works
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
D-PathVer: Dynamic Reasoning Pathways for Complex Claim Verification
Zheng, L., Luo, Z., Hu, W., & Cao, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3349-7_44
FactCellar: An Evidence-based Dataset for Automated Fact-Checking
Dharmavaram, A., Rashid, F. B., & Hakak, S. (2025) · Conference on Privacy, Security and Trust · doi:10.1109/pst65910.2025.11268820
WildClaims: Conversational Information Access in the Wild(Chat)
Joko, H., Amirshahi, S., Clarke, C. L. A., & Hasibi, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_47
IILAP+: Exploring AI-Assisted Dataset Creation for Critical Thinking Assessment
Nurbakova, D., & Магдановна, Е. Л. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811569
FactOWL: A Cost-Efficient Tool for Long-Form Factuality Evaluation
Sakhovskiy, A., Sushko, N., Marina, M., Konovalov, V., Tutubalina, E. V., Panchenko, A., & Braslavski, P. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808373
Post-Generation Audit for Detecting Unsupported Claims in KPI-Grounded LLM Performance Feedback
Belarache, O., Hannouni, S., Ounacer, S., et al. (2026) · International Conference on Intelligent Systems and Digital Applications (ISDA) · doi:10.1109/isda70544.2026.11606099
LEMON: Factor-Level Semantic Alignment for Graph-Text Pairs
Tomilov, A., Gineva, D., Tirskikh, D., Koroteeva, O., & Matveev, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37867-5_25
Claim level graph reasoning for training free LLM hallucination mitigation
Wang, H., & Zhao, W. (2026) · Discover Computing · doi:10.1007/s10791-026-10667-z
An Improved Retrieval-Augmented Generation Agent for UAV Situation Awareness Report Generation
Wang, Z., Wang, G., & Zhao, M. (2026) · Drones · doi:10.3390/drones10100765
A Meta-study of Hallucination Detection Across Provenance, Typology, and Granularities
Belikova, J., & Rykov, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41546-2_17
Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models
Chen, Y., Lang, B., & Wei, Y. (2026) · Mathematics · doi:10.3390/math14203761
MIKA: Source reconstructibility for robust document processing with LLMs: Verification and domain partitioning in enterprise settings
Jaime, A., Gil-Costa, V., Errecalde, M. L., et al. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105257

VeriFastScore: Speeding up long-form factuality evaluation
Rajendhran, R., Zadeh, A., Sarte, M., Li, C., & Iyyer, M. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.491 · Cited by 2 · see citing works
Self-citation
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
Self-citation
Merging facts, crafting fallacies: Evaluating the contradictory nature of aggregated factual claims in long-form generations
Chiang, C.-H., & Lee, H.-Y. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.160 · Cited by 5 · see citing works
Not All Claims Are Equally Risky: FACTOR for Adaptive Verification in Factual Long-Form GenerationPreprint
Hassan, A., Kausar, A., Fatima, S. K., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.22474
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts
Liu, X., Zhang, L., Munir, S., Gu, Y., & Wang, L. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.905 · Cited by 4 · see citing works

Authors: Song, Y., Kim, Y., & Iyyer, M.
Affiliations: University of Massachusetts Amherst (US)
ORCID: Kim, Yekyung
Iyyer, Mohit
Venue: Findings of the Association for Computational Linguistics: EMNLP 2024
Publisher: Association for Computational Linguistics
Published: 2024
Pages: 9447-9474
DOI: 10.18653/v1/2024.findings-emnlp.552
OpenAlex: W4404780948
Cited by: 15
References: 0
Citations by year: 2026: 11 · 2025: 4
FWCI: 4.33
Type: conference paper
Language: en
Open access: hybrid
doi.org
Licence: cc-by
Locations: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: National Science Foundation (2202506, 2046248, 2312949)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: factuality evaluation, long-form text generation, claim extraction, open-weight language models, factual hallucination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

VeriScore: Evaluating the factuality of verifiable claims in long-form text generation

Abstract

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks