CoChat
Sign Up
PreprintCited by 9Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations

Open access
Wanner, Miriam, Azzopardi, Leif, Thomas, Paul, Dan, Soham, Van Durme, Benjamin, Craswell, Nick·2025·arXiv
DOI 10.48550/arxiv.2510.07083arXiv 2510.07083
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important. This results in misleading evaluations when vital information is missing or incorrect as it receives the same weight as peripheral details, raising the question: how can we reliably detect such differences when there are errors in key information? Current approaches that measure factuality tend to be insensitive to omitted or false key information. To investigate this lack of sensitivity, we construct VITALERRORS, a benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information. Using this dataset, we demonstrate the insensitivities of existing evaluation metrics to key information errors. To address this gap, we introduce VITAL, a set of metrics that provide greater sensitivity in measuring the factuality of responses by incorporating the relevance and importance of claims with respect to the query. Our analysis demonstrates that VITAL metrics more reliably detect errors in key information than previous methods. Our dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.

AI snapshot

Written from the paper’s full text.

TL;DR

Existing factuality metrics treat all claims as equally important and are insensitive to missing or false key information. The paper introduces VITAL importance-sensitive metrics and the VITALERRORS benchmark, showing VITAL better detects key-information errors, especially with response-level metrics.

Key findings

  • Existing metrics are largely insensitive to key-information errors. For open-ended queries, FACTSCORE precision was 83.62% normal, 83.58% missing, and 78.84% wrong; NUGGETRECALL stayed within 6% across response types. For single-answer queries, FACTSCORE was 82.58% normal, 82.75% missing, and 76.63% wrong; NUGGETRECALL stayed within 9%.
  • VITAL precision/recall showed larger drops for wrong or missing key information. Single-answer VITAL PREC. fell from 72.81% normal to 48.73% wrong, while VITAL REC. fell from 52.44% normal to 27.45% missing. Open-ended VITAL PREC. was 82.69% normal, 82.88% missing, and 75.40% wrong; VITAL REC. was 40.90% normal, 29.91% missing, and 37.89% wrong.
  • Response-level VITAL metrics best captured adversarial edits. VITAL RLP detected wrong vital subclaims in 73.79% of open-ended wrong responses and 89.53% of single-answer wrong responses, versus 52.70% and 45.03% for normal responses. VITAL RLR detected unsupported vital nuggets in 91.18% of open-ended missing responses and 68.10% of single-answer missing responses, versus 87.22% and 54.30% for normal responses.
  • VITALERRORS contains 6,733 queries from six datasets: FACTSCORE Bios (500), WILDHALLUCINATIONS (2,484), BRIGHT (749), HOTPOTQA (1,000 subset), NQ (1,000 subset), and TRIVIAQA (1,000 subset), covering open-ended and single-answer queries.

Methods

  • The authors constructed VITALERRORS by prompting gpt-4o to generate long-form normal responses to queries, then adversarially altering each response to omit or falsify the single most important piece of information, yielding normal, missing, and wrong response variants.
  • They evaluated FACTSCORE, NUGGETRECALL, and VITAL on these responses. FACTSCORE used gpt-4o for atomic fact generation and evaluation; NUGGETRECALL used AutoNuggetizer with gpt-4o for nugget creation, vital/okay labeling, and support evaluation.
  • VITAL adds an LLM importance-ranking and labeling step for subclaims/nuggets (vital, okay, less important) and computes decomposition-level VITAL PREC./VITAL REC. plus response-level VITAL RLP/VITAL RLR, comparing metric behavior across normal, missing, and wrong responses.

Limitations

  • Stated: The approach depends on LLMs for decomposition, ranking, and verification, so it inherits bias and inconsistencies from the underlying model.
  • Stated: VITAL depends on the quality of retrieved reference material; the authors assume paired sources are factual and relevant, but poor grounding can cause false positives or false negatives, especially where high-quality references are sparse.
  • Stated: VITAL adds an extra importance-ranking LLM call per response, which may limit scalability. The ethics statement also notes automated adversarial perturbations may not capture real-world error diversity, LLM importance judgments may reinforce biases, and the benchmark could be misused in high-stakes contexts. Evident from the analysis: open-ended queries are harder to evaluate because nugget creation depends on provided documents and it is unclear which information is most important, producing many vital claims and more opportunities for errors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.07083

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 24 cited sources checked · 8 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Previous methods for evaluating factual precision employ a decompose-then-verify framework, which involves first decomposing claims from generated responses into atomic subclaims, independently verifying each, and aggregating into a score.

    “Previous methods for evaluating factual precision employ a decompose-then-verify framework, which involves first decomposing claims from generated responses into atomic subclaims”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Min et al., 2023 source
    • cites Song et al., 2024 source
    • cites Wanner et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important.

    “Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important.”
    • Also stated in: § 1 Introduction · plain
      “These factuality metrics treat each piece of information (subclaim) equally, and tend not to explicitly consider whether a claim in the response is vital to answering a user's query.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Min et al., 2023 source

      Section 3.1 (p. 3) explicitly says FACTSCORE assumes every atomic fact has equal weight of importance. That supports the claim for this method, but the paper does not establish that existing factuality-evaluation methods generally treat all claims this way. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Song et al., 2024 source

      Section 2.4 (page unavailable) says VERISCORE adopts SAFE’s F1@K metric and calculates it from counts of supported claims, without assigning different importance weights to those claims. This supports the point for that metric, but the paper does not establish that all existing factuality-evaluation methods treat every claim as equally important; Section 2.1.1 also describes SAFE’s relevance check, which decides whether a claim is worth checking. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Wanner et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    We construct VITALERRORS, a benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information.

    “we construct VITALERRORS, a benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information.”
    • Also stated in: § 3.2 VITALERRORS Dataset · plain
      “Queries We used a total of 6,733 queries from six datasets of varying query type.”
    Quote in the paperSubstantial claim
  • Findingown data

    There is a small difference in VITAL REC. scores for all responses (less than a 5% difference) for open-ended queries.

    “There is a small difference in VITAL REC.scores for all responses (less than a 5% difference) for open-ended queries.”
    Quote in the paperNot a substantial claim
  • Findingown data

    We see a much larger dip of 25% for single-answer queries.

    “We see a much larger dip of 25% for single-answer queries.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 44
  • May combine two claims. 25
  • Not yet classified. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Similar methods are used in evaluating recall, where references are used to generate nuggets-pieces of information vital for answering a given query.

    “Similar methods are used in evaluating recall, where references are used to generate nuggets-pieces of information vital for answering a given query.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Voorhees, 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Pradeep et al., 2024 source

      Sections 3 and 3.1 say the framework generates nuggets from relevant documents and scores responses by whether they contain those nuggets. But Section 3 distinguishes “vital” nuggets from “okay” nuggets, which are useful but not necessary for a good answer; the claim describes all nuggets as vital. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Pradeep et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Godbole and Jia (2025) re-evaluated these evaluation metrics, and find pitfalls in current factuality metrics.

    “Godbole and Jia (2025) re-evaluated these evaluation metrics, and find pitfalls in current factuality metrics.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Godbole and Jia, 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lin and Demner-Fushman (2006) showed how importance can be extended to grades of importance by combing ratings from multiple assessors.

    “Lin and Demner-Fushman (2006) showed how importance can be extended to grades of importance by combing ratings from multiple assessors (i.e., nugget importance is based on annotator popularity).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lin and Demner-Fushman, 2006 source

  • Classified as probably not a substantial claim.

    Premisecitation

    Factuality metrics operate by decomposing an individual response into smaller per-response units, or subclaims, which are then verified against retrieved evidence.

    “In contrast, factuality metrics operate by decomposing an individual response into smaller per-response units, or subclaims.These subclaims are then verified against retrieved evidence.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Min et al., 2023 source

      Section 4.1 (p. 6) says FActScore decomposes a generation into atomic facts and validates them; its Retrieve → LM variant checks those facts using retrieved passages. The same section also describes a no-context variant, so the paper does not establish that factuality metrics generally verify subclaims against retrieved evidence. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Wanner et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Pradeep et al. (2025) implement an automated LLM-based framework using the original vital/okay labels, called AutoNuggetizer, for nugget creation, ranking, assignment, and scoring.

    “Recently, Pradeep et al. (2025) implement an automated LLM-based framework using the original vital/okay labels, called AutoNuggetizer, for nugget creation, ranking, assignment, and scoring (Pradeep et al., 2024).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Pradeep et al., 2025 source
    • cites Pradeep et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The percentage of nuggets supported by the response is the recall score.

    “The percentage of nuggets supported by the response is the recall score (Voorhees, 2003;Pradeep et al., 2024Pradeep et al., , 2025)).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Voorhees, 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Pradeep et al., 2024 source
    • cites Pradeep et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Later mentions of the Apache tribe still remain in the wrong response, however the initial answer is incorrect.

    “Later mentions of the Apache tribe still remain in the wrong response, however the initial answer is incorrect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The second response is missing key information. Because key information is missing, there are no subclaims labeled as vital.

    “The second response is missing key information.Because key information is missing, there are no subclaims labeled as vital.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Using queries from six QA datasets, we generate normal LLM responses, and adversarially alter each response to omit or falsify key information.

    “Using queries from six QA datasets, we generate normal LLM responses, and adversarially alter each response to omit or falsify key information.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    VITAL metrics show moderate improvements over baseline precision and recall, with the response-level evaluations best capturing the impact of key information errors.

    “VITAL metrics show moderate improvements over baseline precision and recall, with the response-level evaluations best capturing the impact of key information errors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    From Table 3, there are fewer nuggets than subclaims (because a nugget often contains multiple subclaims) when comparing the same subset.

    “From Table 3, we first observe there are fewer nuggets than subclaims (because a nugget often contains multiple subclaims) when comparing the same subset (e.g., normal open-ended subclaim total vs. nuggets total)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    We used a total of 6,733 queries from six datasets of varying query type.

    “Queries We used a total of 6,733 queries from six datasets of varying query type.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Despite the answers in the subclaims being wrong, they are labeled as vital subclaims because they are answering exactly what the query is asking.

    “Crucially, despite the answers in the subclaims being wrong, they are labeled as vital subclaims because they are answering exactly what the query is asking.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Through the construction of VITALERRORS dataset, we showed that current methods are insensitive and assign scores to responses of drastically different quality.

    “Through the construction of VITALERRORS dataset, we showed that current methods are insensitive and assign scores to responses of drastically different quality.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    We draw upon FACTSCORE and nuggets to propose a novel importance based factuality score of key information for evaluating responses.

    “We draw upon FACTSCORE and nuggets to propose a novel importance based factuality score of key information for evaluating responses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    We introduce a new set of metrics, which we call VITAL, which includes subclaim ranking and importance labeling for precision metrics.

    “We introduce a new set of metrics, which we call VITAL, which includes subclaim ranking and importance labeling for precision metrics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Responses with missing key information are shorter, and therefore contain fewer subclaims, while responses with wrong key information still have a similar number of subclaims as the original response.

    “Responses with missing key information are shorter, and therefore contain fewer subclaims, while responses with wrong key information still have a similar number of subclaims as the original response.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    We introduced VITAL, a set of metrics designed to evaluate key information factual accuracy and recall.

    “We introduced VITAL, a set of metrics designed to evaluate key information factual accuracy and recall.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For NUGGETRECALL, all responses (normal/missing/wrong) are within 6% of one another for open-ended responses and 9% for single-answer responses, with the highest score for normal responses, and lowest for missing responses.

    “For NUGGETRECALL, we observe all responses (normal/missing/wrong) are within 6% of one another for open-ended responses and 9% for single-answer responses, with the highest score for normal responses, and lowest for missing responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The last response contains the same first subclaim as the normal response, which also is labeled as less important.

    “The last response contains the same first subclaim as the normal response, which also is labeled as less important.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Most of the subclaims are labeled as less important background knowledge, not needed for answering the query.

    “Most of the subclaims are labeled as less important background knowledge, not needed for answering the query.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We observe good detection of incorrect responses here, with 75% (open-ended) and almost 90% (single-answer) of wrong responses getting vital subclaims wrong.

    “We observe good detection of incorrect responses here, with 75% (open-ended) and almost 90% (single-answer) of wrong responses getting vital subclaims wrong.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    VITAL PREC and VITAL REC show a clear difference between normal and adversarial responses, where FACTSCORE and NUGGETRECALL are less sensitive.

    “Table 4 : Precision and recall results for open-ended and single-answer queries.VITAL PREC.and VITAL REC.show a clear difference between normal and adversarial responses, where FACTSCORE and NUGGETRECALL are less sensitive.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Importance judgments for subclaims and nuggets in VITAL are automatic by LLMs.

    “Importance judgments for subclaims and nuggets are automatic by LLMs, which introduces the risk of reinforcing biases present in the model, either under-or over-weighting certain parts of a response, depending on what the model views as”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Responses with wrong key information receive low initial scores, which increases with response length.

    “Similarly to single-answer queries (Figure 3), we observe responses with wrong key information receive low initial scores, which increases with response length.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Normal responses highlight correct vital subclaims, while wrong responses show incorrect but still vital subclaims.

    “Table 9: Example subclaim rankings for a query from NQ.Normal responses highlight correct vital subclaims, while wrong responses show incorrect, but still vital subclaims.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For open-ended queries, wrong responses show low precision early on due to falsified key claims, while normal and missing responses stay consistently high.

    “Figure 4 : Figure 4: Cumulative precision over subclaim position for open-ended queries.Wrong responses show low precision early on due to falsified key claims, while normal and missing responses stay consistently high.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    VITALERRORS covers open-ended and single-answer queries across six sources, totaling 6,733 instances.

    “Table 1 : Overview of datasets used to construct VITALERRORS, covering open-ended and single-answer queries across six sources, totaling 6,733 instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Jiang et al. (2025) extended this work to fix decomposition redundancy, and introduce CORE, a method for subclaim selection based on uniqueness and informativeness.

    “Jiang et al. (2025) extended this work to fix decomposition redundancy, and introduce CORE, a method for subclaim selection based on uniqueness and informativeness.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jiang et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The evaluation of open-ended queries with nuggets is difficult.

    “The evaluation of open-ended queries with nuggets is difficult.”
    Quote in the paperNot a substantial claim

  • Not yet classified.

    Interpretation

    VITAL is dependent on the quality of the retrieved reference material used for verification.

    “Like other factuality metrics, VITAL is dependent on the quality of the retrieved reference material used for verification.”
    Quote in the paperNot yet classified

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    For open-ended queries, it is not clear how to determine what the most important piece of information is.

    “For these open-ended queries, it is not clear how to determine what the most important piece of information is.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    An crucial step in the precision evaluation is the subclaim ranking and importance labeling.

    “An crucial step in the precision evaluation is the subclaim ranking and importance labeling.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Responses to open-ended queries may cover a lot of ground, with an unclear hierarchy to the subclaims within the response.

    “Responses to open-ended queries may cover a lot of ground, with an unclear hierarchy to the subclaims within the response.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Each nugget is assigned an importance label, vital or okay, depending on how important the nugget is for answering the query.

    “Each nugget is assigned an importance label, vital or okay, depending on how important the nugget is for answering the query.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Voorhees, 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Our dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.

    “Our dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Voorhees (2003) introduced nuggets as a minimal, atomic claim that sounds as a correct and useful piece of information in answering a query.

    “Voorhees (2003) introduced nuggets as a minimal, atomic claim that sounds as a correct and useful piece of information in answering a query.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Voorhees, 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    All of these pieces of information tend to be included in a biography.

    “All of these pieces of information tend to be included in a biography.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs are known to produce hallucinations, which can sometimes be harmful.

    “These models are known to produce hallucinations, which can sometimes be harmful (Bender et al., 2021;Dev et al., 2022).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Bender et al., 2021 source
    • cites Dev et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Both frameworks share the intuition of breaking down long-form text into smaller, verifiable units, however they diverge in scope (query-centric vs. response-centric), evidence base (rich pool of candidate material vs. retrieval material), objective (relevance vs. factual correctness), and granularity (nugget vs. claim).

    “Both frameworks share the intuition of breaking down long-form text into smaller, verifiable units, however they diverge in scope (query-centric vs. response-centric), evidence base (rich pool of candidate material vs. retrieval material), objective (relevance vs. factual correctness), and granularity (nugget vs. claim).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    In a human biography, it is not established if a person's occupation, accomplishments, birth place, etc. is more important than the other.

    “For example, in a human biography, it is not established if a person's occupation, accomplishments, birth place, etc. is more important than the other.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The answer need only contain a few pieces of information, which make up the nuggets.

    “The answer need only contain a few pieces of information, which make up the nuggets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Together, VITAL metrics and VITALERRORS offer a new benchmark in factual precision for evaluating and detecting errors in key information.

    “Together, VITAL metrics and VITALERRORS offer a new benchmark in factual precision for evaluating and detecting errors in key information.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Using VITAL with a retriever model adds another variable.

    “In this work, we use sources already paired with the query from previous datasets, and assume this grounding is factual and relevant, but using this framework with a retriever model adds another variable.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The grounding from previous datasets is factual and relevant.

  • May combine two claims.

    Interpretationreasoning

    This results in far more claims labeled as vital, with more possibilities of getting at least one vital subclaim wrong.

    “This results in far more claims labeled as vital, with more possibilities of getting at least one vital subclaim wrong.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Using nuggets to evaluate single-answer queries is quite simple.

    “On the other hand, using nuggets to evaluate singleanswer queries is quite simple.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    These metrics focus on only vital information, and evaluate and detect errors only in this part of the response.

    “These metrics focus on only vital information, and evaluate and detect errors only in this part of the response”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Fewer nuggets means there is less opportunity to make a mistake.

    “Fewer nuggets means there is less opportunity to make a mistake.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Nugget creation is dependent on what documents are provided during the nugget creation process, and therefore to get a high recall score, the generated text should match the content of the provided documents, which is not always likely to happen, especially for entities with diverse masses of information.

    “Nugget creation is dependent on what documents are provided during the nugget creation process, and therefore to get a high recall score, the generated text should match the content of the provided documents, which is not always likely to happen, especially for entities with diverse masses of information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The ranking/importance labeling step in VITAL requires more LLM calls (an extra importance ranking call per response).

    “This step improves metric sensitivity, but requires more LLM calls (an extra importance ranking call per response), which may limit scalability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Poor grounding is particularly difficult in domains where high-quality references are sparse.

    “Poor grounding can cause false positives or false negatives, a challenge that is particularly difficult in domains where high-quality references are sparse.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    These metric changes address issues of equal subclaim weighting and lack of subclaim importance.

    “These metric changes address issues of equal subclaim weighting and lack of subclaim importance.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The nugget framework assumes that a sufficiently broad pool of potentially relevant material can be assembled-typically by pooling results from multiple retrieval systems-and that annotators (or now LLMs) can then extract nuggets representing the key information needs of a query.

    “The nugget framework assumes that a sufficiently broad pool of potentially relevant material can be assembled-typically by pooling results from multiple retrieval systems-and that annotators (or now LLMs) can then extract nuggets representing the key information needs of a query.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Voorhees, 2003

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    In the response-level metrics, there is more opportunity to make a mistake in one of the vital subclaims, explaining the high percentage of responses with any vital nuggets unsupported.

    “In the response-level metrics, this means there is more opportunity to make a mistake in one of the vital subclaims, explaining the high percentage of responses with any vital nuggets unsupported.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Automatic LLM importance judgments may under- or over-weight certain parts of a response depending on what the model views as important.

    “Importance judgments for subclaims and nuggets are automatic by LLMs, which introduces the risk of reinforcing biases present in the model, either under-or over-weighting certain parts of a response, depending on what the model views as”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Automatic LLM importance judgments introduce the risk of reinforcing biases present in the model.

    “Importance judgments for subclaims and nuggets are automatic by LLMs, which introduces the risk of reinforcing biases present in the model, either under-or over-weighting certain parts of a response, depending on what the model views as”
    Quote in the paperNot a substantial claim
56 of 56

  • The paper relies on this without support.

    A different LLM would not yield a different result; gpt-4o is a representative model of the types of models used for evaluation.

    “We are evaluating factuality systems and have no reason to believe a different LLM would yield a different result, and for this reason we pick gpt-4o (OpenAI et al., 2024) as a representative model of the types of models used for evaluation.”

  • The paper relies on this without support.

    The initial answer to the query is the part of the answer which LLM users will pay the most attention to.

    “The initial answer to the query is the part of the answer which LLM users will pay the most attention to.”

  • The authors state this as an assumption.

    The grounding from previous datasets is factual and relevant.

    “In this work, we use sources already paired with the query from previous datasets, and assume this grounding is factual and relevant, but using this framework with a retriever model adds another variable.”

  • The authors state this as an assumption.

    Each subclaim/nugget ranked higher than another is strictly more important.

    “This weighting assumes that each subclaim/nugget ranked higher than another is strictly more important.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Response-level precision is mislabeled as VITAL PREC.

    Minor

    The percentage of responses with any unsupported vital subclaim is defined as VITAL RLP, not VITAL PREC., which measures the fraction of vital subclaims supported. The paragraph describing Table 5 uses the decomposition-level metric name for the response-level result.

  • Response-level recall is mislabeled as VITAL REC.

    Minor

    The percentage of responses with any unsupported vital nugget is defined as VITAL RLR. VITAL REC. instead measures the fraction of vital nuggets supported, so the Table 5 discussion names the wrong metric.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Ellen M. Voorhees. 2003. Overview of the TREC 2003 question answering track. In Proceedings of The Twelfth Text REtrieval Conference, TREC 2003, Gaithersburg, Maryland, USA, November 18- 21, 2003, volume 500-255 of NIST Special Publica- tion, pages 54-68. National Institute of Standards and Technology (NIST).

Cited by

Sorted by citations

Related

Authors
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N.
Affiliations
  • Microsoft (United States) (US)
  • University of Strathclyde (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.07083
arXiv
2510.07083
OpenAlex
W4415318059
Cited by
0
References
25
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
factuality evaluation, large language models, omitted information

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review