CoChat
Sign Up
PreprintCited by 52Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Long-form factuality in large language models

Open access
Wei, Jerry, Yang, Chengrun, Song, Xinying, Lu, Yifeng, Hu, Nathan, Huang, Jie, Tran, Dustin, Peng, Daiyi +4 more·2024·arXiv
DOI 10.48550/arxiv.2403.18802arXiv 2403.18802
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall). Empirically, we demonstrate that LLM agents can outperform crowdsourced human annotators - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces LongFact, a 2,280-prompt benchmark for long-form factuality across 38 topics, and SAFE, an LLM-agent evaluator that decomposes responses into facts and verifies them via Google Search. SAFE agrees with crowdsourced human annotations on 72.0% of 16,011 individual facts and wins 76% of a 100-case disagreement sample while costing over 20× less.

Key findings

  • LongFact contains 2,280 fact-seeking prompts across 38 manually selected topics, split into LongFact-Concepts and LongFact-Objects (1,140 prompts each).
  • On 16,011 individual facts, SAFE agrees with crowdsourced human annotators 72.0% of the time; on a random subset of 100 disagreement cases, SAFE is correct 76% of the time versus 19% for human annotators, using “researcher + full internet access” labels as ground truth.
  • SAFE costs $0.19 per model response versus $4.00 for crowdsourced human annotations, making it more than 20× cheaper.
  • Benchmarking 13 LLMs across Gemini, GPT, Claude, and PaLM-2 on 250 random LongFact-Objects prompts found larger models generally achieve better long-form factuality; at K=64 and K=178, the top three models are GPT-4-Turbo (F1@64 95.0, F1@178 66.4), Gemini-Ultra (91.7, 60.3), and PaLM-2-L-IT-RLHF (91.0, 55.3).
  • Including recall in F1@K correlates better with Chatbot Arena ELO (r=0.754, p=0.031) than precision alone (r=0.502, p=0.205).

Methods

  • LongFact was generated by prompting GPT-4 with 38 manually selected topics, producing 30 unique prompts per topic per task after deduplication.
  • SAFE uses an LLM (GPT-3.5-Turbo) to split a response into individual facts, revise them to be self-contained, judge relevance, and then issue up to five Google Search queries per fact via the Serper API, keeping three results per query, before labeling each fact supported, not supported, or irrelevant.
  • The benchmark evaluated 13 models on 250 random LongFact-Objects prompts, decoding up to 1,024 tokens at temperature zero, then aggregating SAFE outputs with F1@K at K=64 (median relevant facts) and K=178 (maximum relevant facts).

Limitations

  • Stated in the paper: LongFact and SAFE depend on LLM capabilities, so weak instruction-following or reasoning can degrade prompt generation and fact rating; SAFE relies on Google Search as a knowledge source, which may fail in corner cases or expert domains such as law and medicine; F1@K assumes responses contain no repeated facts and can be gamed by repeating a supported fact.
  • Stated in the paper: SAFE outperforms crowdsourced human annotators but does not necessarily match expert-level humans; LongFact-Concepts was excluded from the main benchmark because it is easier than LongFact-Objects.
  • Evident from the text/checklist: the main benchmark reports no error bars or statistical significance tests, and the paper does not report compute resources for experiments because many rely on proprietary models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.18802

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

18 of 30 cited sources checked · 12 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Large language models often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics.

    “Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Huang et al., 2023 source

      Section 2.3 describes factual errors in LLM answers to fact-seeking questions, and the Introduction describes a tendency to generate factually unsupported content. But Section 2.3 says LLMs exhibit this tendency “occasionally”; it does not support the claim that they “often” do so when answering open-ended prompts. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Zhang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Google Search can be leveraged to obtain dynamic reference answers for each fact in a response with the best tradeoff among accuracy, accessibility, and timeliness.

    “Google Search can be leveraged to obtain dynamic reference answers for each fact in a response with the best tradeoff among accuracy, accessibility, and timeliness (Augenstein et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Augenstein et al., 2023 source

      The Introduction (page unavailable) says search engines such as Google are more reliable than ChatGPT because they clearly indicate their sources. It does not say Google Search provides dynamic reference answers for each fact or offers the best tradeoff among accuracy, accessibility, and timeliness. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    Google Search holds a large amount of information about many subjects that allows it to serve as a strong proxy for factual accuracy.

    “Google Search holds a large amount of information about many subjects that allows it to serve as a strong proxy for factual accuracy (Augenstein et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Augenstein et al., 2023 source

      The Introduction (page unavailable) describes search engines such as Google as historically reliable gateways to authoritative sources, while noting that they have sometimes amplified disinformation. It does not say that Google Search's breadth of information makes it a strong proxy for factual accuracy; the Unreliable Evaluation section instead describes maintaining factuality ground truth as an open problem. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Min et al. (2023) reported a cost of $4 per model response to obtain crowdsourced human annotations for the dataset.

    “Min et al. (2023) reported a cost of $4 per model response to obtain crowdsourced human annotations for the dataset.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al. (2023) source
  • Premisecitation

    Other metrics such as token-level F1 score, sentence or passage-level AUC, and similarity to ground-truth can account for recall as well as precision but require human responses or judgements that serve as a ground-truth.

    “Other metrics such as token-level F 1 score (Rajpurkar et al., 2016), sentence or passage-level AUC (Manakul et al., 2023), and similarity to ground-truth (Wieting et al., 2022) can account for recall as well as precision but require human responses or judgements that serve as a ground-truth.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Rajpurkar et al., 2016 source
    • cites Manakul et al., 2023 source

      Table 2 reports sentence-level AUC-PR, a precision–recall measure, and §6 describes the human-annotated labels used as ground truth. But Table 2 measures passage-level performance with Pearson and Spearman correlations, not AUC; the passage-level AUC part of the claim is unsupported. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Wieting et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. 55
  • Classified as probably not a substantial claim. 58

  • May combine two claims.

    Premisecitation

    The annotations from Min et al. (2023) were gathered from crowdsourced human raters and are not at the level of expert-level humans.

    “These annotations, however, were gathered from crowdsourced human raters, and thus are not at the level of expert-level humans.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

      The Data section (p. 4) confirms that the annotators were recruited as freelancers through Upwork. But Appendix A.3 (p. 15) says the authors recruited freelancers who listed fact-checking as their expertise and that every worker passed a two-hour qualification test. This contradicts the claim that the annotations were not at an expert level. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationown data

    SAFE outperforms crowdsourced human annotators, agreeing with 72% of human annotations from Min et al. (2023) and winning 76% of disagreement cases from a random sample of 100.

    “Empirically, SAFE outperforms crowdsourced human annotators, agreeing with 72% of human annotations from Min et al. (2023) and winning 76% of disagreement cases from a random sample of 100 disagreement cases”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Long-form responses should be evaluated at the granularity of individual facts.

    “we agree with the idea that long-form responses should be evaluated at the granularity of individual facts, following the text decomposition in Min et al. (2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

  • May combine two claims.

    Findingcitation

    Rating a single model response costs $4.00 using human annotations, compared to $0.19 using SAFE configured with GPT-3.5-Turbo and Serper API.

    “rating a single model response costs $4.00 using human annotations(Min et al., 2023), compared to $0.19 using SAFE configured with GPT-3.5-Turbo and Serper API.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Tian et al. (2023) and Dhuliawala et al. (2023) also use precision as a proxy for a model response's factuality.

    “Tian et al. (2023) and Dhuliawala et al. (2023) also use precision as a proxy for a model response's factuality.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Tian et al., 2023 source
    • cites Dhuliawala et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    FActScore measures precision with respect to Wikipedia pages as knowledge sources and mentions the difficulty of measuring recall as future work.

    “FActScore (Min et al., 2023) measures precision with respect to Wikipedia pages as knowledge sources and mentions the difficulty of measuring recall as future work.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Min et al., 2023 source

  • Classified as probably not a substantial claim.

    Premisecitation

    Min et al. (2023) includes irrelevant facts when computing the total number of facts in a response.

    “This diverges from Min et al. (2023), which includes irrelevant facts when, for example, computing the total number of facts in a response.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Prior work (Min et al., 2023) has noted incorporating recall when measuring long-form factuality but has not thoroughly explored it.

    “This idea of incorporating recall when measuring long-form factuality has been noted by prior work (Min et al., 2023) but has not been thoroughly explored.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

  • May combine two claims.

    Findingown data

    On a random subset of 20 questions from the FActScore biography dataset, among 146 differently-rated claims among all 657, there were two cases where the difference is attributed to search scope: no-scope wins and Wikipedia-only wins.

    “On a random subset of 20 questions from FActScore biography dataset (Min et al., 2023), we manually examine the 146 differently-rated claims among all 657.We found the following two cases where the difference is attributed to search scope:”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Min et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    Crowdsourced human annotators from Min et al. (2023) incorrectly rated an individual fact because the human rater was not given the necessary information to correctly rate the fact, as the correct information is located on a different Wikipedia page.

    “The presumed cause of error was that the human rater was not given the necessary information to correctly rate the fact because the correct information is located on a different Wikipedia page.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Min et al. (2023) source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    SAFE issued GPT-3.5-Turbo API calls costing $64.57 and Serper API calls costing $31.74, thus costing a total of $96.31, equivalent to $0.19 per model response.

    “SAFE issued GPT-3.5-TurboAPI calls costing $64.57 and Serper API calls costing $31.74, thus costing a total of $96.31, equivalent to $0.19 per model response.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On disagreement cases, SAFE annotations were correct 76% of the time, whereas human annotations were correct only 19%.

    “We found that on these disagreement cases, SAFE annotations were correct 76% of the time, whereas human annotations were correct only 19%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results.

    “SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The maximum number of relevant facts among tested model responses is 178.

    “We thus also present results in Section 6 for K = 178, the maximum number of relevant facts among tested model responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Thirty unique prompts per topic were generated for a total of 1,140 prompts per task.

    “We generate 30 unique prompts per topic for a total of 1,140 prompts per task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    LongFact consists of two tasks, LongFact-Concepts and LongFact-Objects, separated based on whether the questions ask about concepts or objects.

    “LongFact consists of two tasks, LongFact-Concepts and LongFact-Objects, separated based on whether the questions ask about concepts or objects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    K=48 is the 25th percentile number of facts in a response out of all model responses.

    “K = 48 (the 25th percentile number of facts in a response out of all model responses)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There were minor causes of human annotator error, including failing to carefully read the model response and incorrectly reasoning that a piece of information supports a fact when it actually does not.

    “There were also minor causes of error, including the annotator failing to carefully read the model response (e.g., labeling a supported fact about the subject of the prompt as”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    At higher values of K, including the postamble yields responses that are more factual, indicating higher recall.

    “Additionally, we find that at higher values of K, including the postamble yields responses that are more factual, indicating that these responses have higher recall (i.e., have more supported facts), which is expected since the postamble specifically asks the model to provide as many facts as possible.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    LongFact includes topics from MMLU (Hendrycks et al., 2021) for broader topic coverage, as well as topics such as celebrities, gaming, movies, and music to account for less academic topics.

    “LongFact includes topics from MMLU (Hendrycks et al., 2021) for broader topic coverage, as well as topics such as celebrities, gaming, movies, and music, which we added in an effort to account for topics that are less academic in nature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hendrycks et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    LongFact includes 38 manually selected topics to provide broad coverage of long-form factuality across many topics.

    “We manually selected a set of 38 topics for LongFact in order to provide broad coverage of long-form factuality across many topics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The second-largest error occurred because the language model did not issue search queries that would find the necessary information or because the relevant Google Search result was not a top result.

    “This occurs because the language model did not issue search queries that would find the necessary information or because the relevant Google Search result was not a top result.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4-Turbo is better than GPT-4, which is better than GPT-3.5-Turbo.

    “GPT-4-Turbo is better than GPT-4 which is better than GPT-3.5-Turbo”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    K=84 is the 75th percentile number of facts in a response out of all model responses.

    “K = 84 (the 75th percentile number of facts in a response out of all model responses)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On a random subset of 20 questions from the FActScore biography dataset, among all 658 atomic facts, Mixtral 8×7B wins 1/3 of the differently-rated facts while GPT-3.5-Turbo wins 2/3.

    “Among all 658 atomic facts, we manually examine the 196 differently-rated facts by the above two settings.Mixtral 8×7B wins 1/3 of the time while GPT-3.5-Turbowins 2/3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    LongFact contains 1,140 prompts for LongFact-Concepts and 1,140 prompts for LongFact-Objects.

    “This results in a total of 1,140 prompts for LongFact-Concepts and 1,140 prompts for LongFact-Objects.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    SAFE obtains high correlation with human annotators for the number of not-supported facts in a response, albeit lower than for supported facts.

    “SAFE also obtains high correlation with human annotators for the number of not-supported facts in a response, albeit lower correlation than the number of supported facts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In more nuanced domains, it is more challenging for SAFE to break down a long sentence into atomic facts and to revise a fact to be self-contained than to rate whether an atomic fact is supported by search results; over 90% of incorrect ratings fall into the former two categories.

    “Our finding is that in more nuanced domains, it is more challenging for SAFE to (1) break down a long sentence into atomic facts and (2) revise a fact to be self-contained, than to rate whether an atomic fact is supported by search results: over 90% of the incorrect ratings fall into the former two categories.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At sufficiently-large K, model rankings are relatively constant; only at K=1 do model rankings significantly change.

    “As shown in Figure 17, we find that at sufficiently-large K, model rankings are relatively constant; only at K = 1 do model rankings significantly change.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On the full set of 496 prompt-response pairs, SAFE's number of individual facts outputted correlates with human-annotated number of individual facts: Pearson r = 0.798 (p = 1.2 × 10^-110) and Spearman r = 0.846 (p = 2.7 × 10^-137).

    “on the full set of 496 prompt-response pairs, SAFE obtains a response-level Pearson correlation of r = 0.798 (p = 1.2 × 10 -110 ) and a Spearman correlation of r = 0.846 (p = 2.7 × 10 -137 ) with human annotations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When only using precision, a model's average performance on LongFact and its arena ELO achieve a Pearson correlation of r = 0.502 (p = 0.205), which is not statistically significant.

    “When only using precision, a model's average performance on LongFact and its arena ELO only achieve a Pearson correlation of r = 0.502 (p = 0.205), which indicates that there is not a statistically-significant correlation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The average number of supported facts in a response from PaLM-2-L-IT-RLHF is 72.9, compared to 13.2 supported facts per response from PaLM-2-L-IT.

    “Indeed, as shown in Table 30, the average number of supported facts in a response from PaLM-2-L-IT-RLHF is 72.9, compared to only 13.2 supported facts per response from PaLM-2-L-IT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For GPT-4-Turbo on LongFact-Objects, instructing the model to give longer responses increases factual recall (number of supported facts) but also increases the proportion of factual errors (lower precision).

    “As we instruct the model to give longer responses, we see an increase in factual recall (i.e., the number of supported facts), but also a larger proportion of factual errors (i.e., lower precision).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    LongFact consists of 38 topics categorized into four supercategories.

    “LongFact consists of 38 topics (listed in Appendix B.2) categorized into four supercategories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The dataset was not scraped from the Internet and was instead generated by a language model.

    “Our dataset was not scraped from the Internet and was instead generated by a language model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For the example of Pearl White being a movie actress, the presumed cause of the human annotator's error was that the human rater did not read the response to see the relationship between Anna May Wong and Pearl White.

    “The presumed cause of error was that the human rater did not read the response to see the relationship between Anna May Wong (the subject of the prompt) and Pearl White (the subject of the individual fact).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    For the example of Sajid Nadiadwala's birthplace, the presumed cause of the human annotator's error was that the human rater missed the information available on the reference Wikipedia page.

    “The presumed cause of error was that the human rater missed the information that was available on the reference Wikipedia page.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    SAFE outperforms human annotators while being more than 20× cheaper.

    “SAFE outperforms human annotators while being more than 20× cheaper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Responses generated by GPT-4-Turbo for long-form factuality prompts have slightly lower factual precision.

    “byGPT-4-Turbo for long-form factuality prompts have slightly-lower factual precision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    For the example of Egon Krenz being released after serving four years, the presumed cause of the human annotator's error was that the human rater missed the technicality that serving almost four years is not the same as serving four years.

    “The presumed cause of error was that the human rater missed the technicality that serving almost four years is not the same as serving four years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The cause of error was that SAFE was unable to find the supporting information through Google Search.

    “The cause of error was that SAFE was unable to find the supporting information through Google Search.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    FActScore may require long-form responses but does not cover a broad range of topics.

    “Other factuality datasets such as FActScore (Min et al., 2023) may require long-form responses but do not cover a broad range of topics.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Min et al., 2023 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The key innovation in SAFE is using a language model as an agent to generate multi-step Google Search queries and carefully reason about whether facts are supported by search results.

    “We consider the key innovation in SAFE to be using a language model as an agent to generate multi-step Google Search queries and carefully reason about whether facts are supported by search results”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    F1 score can be extended as an aggregated metric for long-form factuality by balancing the percentage of supported facts (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall).

    “Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality.To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is some number of supported facts K for which a user cares about recall up to the Kth supported fact.

  • May combine two claims.

    Premisecitation

    Existing factuality benchmarks mostly consist of prompts that test knowledge of a single factoid and thus only require a short response rather than several paragraphs to be fully answered.

    “many established benchmarks such as TruthfulQA (Lin et al., 2022), HaluEval (Li et al., 2023), FreshQA (Vu et al., 2023), HalluQA (Cheng et al., 2023b), and FELM (Chen et al., 2023) mostly consist of prompts that test knowledge of a single factoid”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Lin et al., 2022 source

      Not checked in the free run.

    • cites Li et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Vu et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    LongFact is the first prompt set for evaluating long-form factuality in a wide variety of domains.

    “To our knowledge, LongFact is the first prompt set for evaluating long-form factuality in a wide variety of domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    A changing set of in-context exemplars helps increase prompt diversity.

    “(a changing set of in-context exemplars helps increase prompt diversity)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The configuration of SAFE uses GPT-3.5-Turbo, which provides strong performance in most cases at low cost.

    “our configuration of SAFE uses GPT-3.5-Turbo, which provides strong performance in most cases at low cost”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Google Search may not easily find information about some fact or may lack profundity in expert-level domains such as law and medicine.

    “Google Search may not easily find information about some fact or may lack profundity in expert-level domains such as law and medicine.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    SAFE does not assess the global accuracy of a fact; it may miss supporting information for a factually-accurate fact or find supporting information for a factually-inaccurate fact.

    “We use the”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Min et al. (2023) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Two of the models are internal (PaLM-2-L-IT and PaLM-2-L-IT-RLHF) and therefore inaccessible.

    “While two of our models are internal (namely, PaLM-2-L-IT and PaLM-2-L-IT-RLHF) and therefore inaccessible”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    F1@1 only depends on the precision of a response and is monotonically increasing.

    “Thus, we see that F 1 @1 only depends on Prec(y), the precision of a response, and is monotonically increasing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A step of removing duplicated facts can be added into SAFE.

    “we also suggest that a step of removing duplicated facts can be added into SAFE”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Mixtral 8×7B may have weaker instruction-following capabilities, which could be improved by further post-training or prompt tuning.

    “This disadvantage seems to suggest it may have weaker instruction-following capabilities, which could be improved by further post-training or prompt tuning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    SAFE outperforms human raters' performance with respect to crowdsourced humans, but not necessarily with respect to all humans.

    “We thus note that SAFE outperforms human raters' performance with respect to crowdsourced humans, but not necessarily with respect to all humans.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Users almost certainly do not see a response as more creative after a certain response length is achieved.

    “Similar to long-form factuality, however, users almost certainly do not see a response as more creative after a certain response length is achieved.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Malaviya et al. (2024) contains expert-level annotations for factuality, but the questions used to obtain responses are often not open-ended or not purely fact-seeking.

    “Malaviya et al. (2024) contains expert-level annotations for factuality, but the questions used to obtain responses are often not open-ended or not purely fact-seeking”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Malaviya et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    SAFE can be easily tailored to searching over more restricted and predetermined sources by adding site:xxx to the Google Search query or replacing Serper with another API that queries professional databases.

    “SAFE can be easily tailored to searching over more restricted and predetermined sources by adding”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Using harder prompts amplifies differences in long-form factuality between models because less-factual models have more noticeably lower performance on harder prompts.

    “Thus, we only use LongFact-Objects in order to make the prompt set as difficult as possible and therefore amplify differences in long-form factuality between models since less-factual models should have more-noticeably lower performance when tested on harder prompts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The superior performance of SAFE demonstrates that language models can be used as scalable autoraters that have the potential to outperform crowdsourced human annotators.

    “Overall, the superior performance of SAFE demonstrates that language models can be used as scalable autoraters that have the potential to outperform crowdsourced human annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    LongFact and SAFE rely on LLMs to function, hence the used-LLM's capabilities directly affect the quality of the generated LongFact prompts and SAFE.

    “Both LongFact and SAFE rely on LLMs to function, hence the used-LLM's capabilities (especially instruction following and reasoning, but also abilities such creativity) directly affect the quality of the generated LongFact prompts and SAFE.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Irrelevant facts neither improve nor reduce the factuality of a model's response to a prompt.

    “we regard irrelevant facts as neither improving nor reducing the factuality of a model's response to a prompt.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    F1@K measures both precision and recall of a long-form response without requiring any preset ground-truth answers, only the number of facts from each label category.

    “Our proposed F 1 @K, on the other hand, measures both precision and recall of a long-form response without requiring any preset ground-truth answers, only the number of facts from each label category.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    All models in Section 6 were evaluated by generating responses at a temperature of zero to decrease variation in responses across runs.

    “We also evaluated all models in Section 6 by generating responses at a temperature of zero to decrease variation in responses across runs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Computing F1 at K = T × W × F only allows a response to achieve a score of 1 when it contains as many supported facts as can be fit in the output sequence length and when all provided facts are supported.

    “Computing F 1 at K = T × W × F thereby only allows a response to achieve a score of 1 when it contains as many supported facts as can be fit in the output sequence length and when all provided facts are supported.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    A weak language model may decompose a long-form response into individual facts that contain more than one factual statement, exhibit poor reasoning when determining relevance, propose unrelated search queries, or fail to reason about whether a fact is supported by search results.

    “a weak language model may (a) decompose a long-form response into individual facts that contain more than one factual statement, (b) exhibit poor reasoning when determining the relevance of an individual fact to the prompt, (c) propose unrelated search queries to verify a fact, or (d) fail to reason about whether a fact is supported by search results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    SAFE is automatic, does not require any model finetuning or any preset reference answers/knowledge sources, and is demonstrably more-reliable than crowdsourced human annotations while only requiring a fraction of the cost.

    “SAFE is automatic, does not require any model finetuning or any preset reference answers/knowledge sources, and is demonstrably more-reliable than crowdsourced human annotations while only requiring a fraction of the cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Existing benchmarks either cover singular short factoids or only cover a small set of long-form topics.

    “existing benchmarks either cover singular short factoids or only cover a small set of long-form topics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    LongFact contains prompts that are likely to be used to benchmark and evaluate future large language models.

    “LongFact contains prompts that are likely to be used to benchmark and evaluate future large language models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisereasoning

    Google Search allows access to the entire internet and is arguably the most-comprehensive knowledge source for open-domain factuality tasks that is also widely-accessible.

    “Google Search allows access to the entire internet and thus is arguably the most-comprehensive knowledge source for open-domain factuality tasks that is also widely-accessible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Language models are likely more knowledgeable about general topics than specific entities.

    “language models are likely more knowledgeable about general topics than specific entities”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Model snapshots (e.g., gpt-4-0613) were used when possible to improve reproducibility.

    “when possible, we used model snapshots (e.g., gpt-4-0613) to improve reproducibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A response that contains specific details is likely to be less precise because details are more difficult to get correct than general facts.

    “which follows the intuition that a response that contains specific details is likely to be less precise since details are more-difficult to get correct than general facts”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    F1@K allows for standardized and quantifiable comparisons between language models and between methods of obtaining responses.

    “we contend that it allows for standardized and quantifiable comparisons between language models and between methods of obtaining responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Recall should be included in long-form factuality measurement because when factual precision of two responses is equal, the response with more supported facts should be considered more factual.

    “We contend that recall should be included because when the factual precision of two responses are equal, one intuitively expects that the response that provided more supported facts should be considered more-factual in a longform setting.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Individual facts should be self-contained when rated.

    “When individual facts are rated, they should be self-contained (Gao et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gao et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    SAFE relies on Google Search as a knowledge source to obtain ground-truths, which may not suffice in corner cases.

    “There also exists an inherent weakness in SAFE, which is that it relies on Google Search as a knowledge source to obtain ground-truths, which may not suffice in corner cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The postamble encourages models to provide more facts in their responses and to make facts more-specific.

    “These findings indicate that the postamble indeed encourages models to provide more facts in their responses and to make facts more-specific.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    LongFact-Concepts is inherently easier than LongFact-Objects because it asks about general concepts rather than specific entities.

    “These findings are somewhat expected, however, since LongFact-Concepts is inherently easier than LongFact-Objects as it asks about general concepts (e.g., theories, doctrines, etc.) rather than specific entities (language models are likely more knowledgeable about general topics than specific entities).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    F1@K is bounded between 0 and 1, where a higher number indicates a more-factual response.

    “This proposed metric is bounded between [0, 1], where a higher number indicates a more-factual response.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    It is impossible to come up with a definite set of facts that should be included in a long-form response.

    “Measuring recall is more challenging, however, as it is impossible to come up with a definite set of facts that should be included in a long-form response”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Repetition of facts can be better measured by other metrics such as fluency or usefulness of model responses.

    “we contend that repetition of facts can be better measured by other metrics such as fluency or usefulness of model responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The method of measuring recall with a hyperparameter K is not limited to long-form factuality but can be used in other long-form domains where a response can be infinitely long yet users do not desire longer responses past some point.

    “We contend that this method of measuring recall is not limited to long-form factuality, but can even be used in other long-form domains where a response can be infinitely-long, yet users do not desire longer responses past some point.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The reasoning errors in SAFE can be mostly resolved by using a model with stronger reasoning abilities, such as GPT-4 or GPT-4-Turbo.

    “Because we used GPT-3.5-Turbo,we posit that these reasoning errors can be mostly resolved by using a model that has stronger reasoning abilities, such as GPT-4 or GPT-4-Turbo.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    For a response to receive a full score of 1, it must not contain not-supported facts and must have at least K supported facts.

    “For a response to receive a full score of 1, it must not contain not-supported facts and must have at least K supported facts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    This method of controlling the response length only works for language models that have strong instruction-following capabilities, as is the case with GPT-4-Turbo.

    “This method of controlling the response length only works for language models that have strong instructionfollowing capabilities, as is the case with GPT-4-Turbo.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The dataset was manually reviewed to ensure its quality.

    “We also manually reviewed the dataset to ensure its quality”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A canary string is provided to prevent misuse of the dataset in language-model pretraining.

    “we provide a canary string to prevent misuse of our dataset in language-model pretraining in Appendix A.11.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The prompts used to evaluate the models in Section 6 do not provide information on desired response length, so the results in Section 6 can be viewed as measuring the 'default behavior' of models.

    “The prompts used to evaluate these models, however, do not provide any information on the desired response length, so the results in Section 6 can be viewed as measuring the”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    This rescaling does not change the relative ranking of responses by our metric.

    “This rescaling does not change the relative ranking of responses by our metric.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    SAFE incorrectly rated the individual fact 'Mark Webber is a director' as not supported because it correctly found Google Search results indicating that Mark Webber is a director, but made a reasoning error that because he is primarily known as an actor, he is not a director.

    “The cause of error was that SAFE correctly found Google Search results indicating that Mark Webber is a director, but made a reasoning error that because he is primarily known as an actor, he is not a director.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The precision-recall trade-off observed in long-form factuality mirrors the precision-recall trade-off in a traditional classification setting where the classification threshold is varied.

    “This mirrors the precision-recall trade-off in a traditional classification setting where the classification threshold is varied.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Crowdsourced human annotators from Min et al. (2023) incorrectly rated an individual fact because the human rater confused the 'not-supported' label as 'irrelevant.'

    “The presumed cause of error was that the human rater confused the”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Min et al. (2023) source

      Not checked in the free run.

93 of 93

  • The authors state this as an assumption.

    There is some number of supported facts K for which a user cares about recall up to the Kth supported fact.

    “we posit that there is some number of supported facts K for which a user cares about recall up to the Kth supported fact.”

  • The authors state this as an assumption.

    A response does not contain any repeated facts.

    “it operates under the assumption that a response does not contain any repeated facts.”

  • The authors state this as an assumption.

    The human annotator misidentified a fact as irrelevant when it was actually not supported.

    “We assume that the human annotator then misidentified the fact as being irrelevant when it was actually not supported.”

  • The authors state this as an assumption.

    A given model response does not contain repeated facts.

    “Our implementation of F 1 @K assumes that a given model response does not contain repeated facts”

  • The authors state this as an assumption.

    The evaluation method functions perfectly in the case analysis.

    “In these cases, we assume that the evaluation method functions perfectly and instead focus on analyzing how F 1 @K aggregates the raw metrics from the evaluator into a single number.”

  • The authors state this as an assumption.

    K is a finite number greater than one, meaning that a user cares about having more supported facts up to K supported facts.

    “Assume we set K as some finite number greater than one, meaning that we assume that a user cares about having more supported facts up to K supported facts.”

  • The paper relies on this without support.

    A model response that solely contains one-hundred not-supported facts should be equally factual as a model response that comprises a singular not-supported fact.

    “A model response that solely contains one-hundred not-supported facts should be equally factual as a model response that comprises a singular not-supported fact because both responses fail to provide any factual information.”

  • The paper relies on this without support.

    A model response consisting of a single supported fact should be less factual than a model response that solely contains K supported facts because the response with more supported facts provides more factual information.

    “a model response consisting of a single supported fact should be less factual than a model response that solely contains K supported facts because the response with more supported facts provides more factual information (i.e., has higher recall).”

  • The authors state this as an assumption.

    Newer language model subfamilies have been scaled up for some scaling factor.

    “we generally assume that newer language model subfamilies have been scaled up for some scaling factor”

  • The authors state this as an assumption.

    All Claude-3 models are scaled further than all Claude-2 models.

    “(e.g., we assume that all Claude-3 models are scaled further than all Claude-2 models)”

  • The authors state this as an assumption.

    A response does not contain repeating facts.

    “Our measurement of recall assumes that a response does not contain repeating facts (see Appendix D.1).”

  • The paper relies on this without support.

    A language model is capable of generating a variety of responses to a question given different instructions.

    “While it is important to evaluate this default behavior, a language model is also capable of generating a variety of responses to a question given different instructions.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SAFE's generalizability is questionable given that it is only evaluated on FACTSCORE's biography data.

    “Lastly, SAFE's generalizability is questionable given that it is only evaluated on FACTSCORE's biography data.”

    From VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024) · cites this paper as Wei et al., 2024

    Appendix C.10, “Generalization to more professional domains” (page unavailable), says the authors evaluated SAFE on law and medicine facts from six additional questions and reports agreement rates for both domains. The paper therefore contradicts the assertion that SAFE was evaluated only on FACTSCORE’s biography data. Whether those evaluations fully establish generalizability is a separate question. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Long-form question answering relies heavily on retrieval-augmented generation to generate comprehensive and factual responses.

    “Long-form Question Answering relies heavily on Retrieval-Augmented Generation (RAG) to generate comprehensive and factual responses (Wei et al., 2024;Chen et al., 2025).”

    From MIRAGE: Defending Long-Form RAG Against Misinformation Pollution (2026) · cites this paper as Wei et al., 2024

    The LongFact description (section label ✘; page unavailable) says its questions require long-form responses containing multiple detailed factoids. Related work (section 7; page unavailable) mentions evaluating responses from a RAG system, but the paper does not say that long-form question answering relies heavily on RAG to produce comprehensive, factual answers. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • All four long-form benchmarks used (LongFact, FAVA, AlpacaFact, Biography) are prompt-only, i.e., questions without an accompanying gold evidence set.

    “All four benchmarks are prompt-only (questions without an accompanying gold evidence set).”

    From MIRAGE: Defending Long-Form RAG Against Misinformation Pollution (2026) · cites this paper as Wei et al., 2024

    Section 2 describes LongFact as a set of question prompts, supporting that part of the claim. The paper does not establish that FAVA or AlpacaFact are prompt-only; its Related work section instead says biography prompts are evaluated against each person's Wikipedia article, so it does not support the claim for all four benchmarks. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The authors evaluate factuality with VeriScore F1@k (median over prompts), which decomposes each generated answer into claims and uses a verifier to label each claim as supported, unsupported, or insufficient given evidence snippets.

    “We evaluate factuality with VeriScore (Song et al., 2024) F1@k (median over prompts), which decomposes each generated answer into claims and uses a verifier to label each claim as supported, unsupported, or insufficient given evidence snippets (Wei et al., 2024); see Appendix E.”

    From MIRAGE: Defending Long-Form RAG Against Misinformation Pollution (2026) · cites this paper as Wei et al., 2024

    The SAFE section describes breaking responses into individual facts and using search results to judge whether each is supported. But its labels are supported, not supported, and irrelevant—not supported, unsupported, and insufficient. The paper’s benchmarking table also says its metrics are averaged over prompts, not reported as a median. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Existing metrics FACTSCORE and SAFE decompose input text into atomic claims and verify each against a knowledge base like Wikipedia.

    “Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into”

    From VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024) · cites this paper as Wei et al., 2024

    The SAFE section and Appendix C.4 say that SAFE splits responses into individual facts and checks them against search results, using a splitting step almost identical to Min et al. (2023). The Related Work section says FActScore uses Wikipedia pages as knowledge sources. But SAFE verifies facts through iterative Google Search queries, not against a Wikipedia-like knowledge base, so that part of the claim does not accurately describe SAFE. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Checklist denies releasing a new dataset

    Minor

    The ethics checklist says the paper does not release new datasets, but the new-assets checklist explicitly says it releases one. The paper also identifies LongFact as a publicly available prompt set.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Conference of the As- sociation for Computational Linguistics, 2004. URL https://aclanthology.org/W04-1013.
  2. Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023. URL https: //arxiv.org/abs/2309.09558.

Cited by

Sorted by citations

Related

Authors
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Huang, J., Tran, D., Peng, D., Liu, R., Huang, D., Du, C., & Le, Q. V.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
80756-80827
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2403.18802
arXiv
2403.18802
OpenAlex
W4415797127
Semantic Scholar
58e194695187fe4daeb04ea694e0f59af2441177
Cited by
36
References
0
Citations by year
2026: 26 · 2025: 6 · 2024: 4
FWCI
10.39
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, factual consistency, long-form text generation, hallucinations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review