CoChat
Sign Up
PreprintCited by 2Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

Open access
Haas, Lukas, Yona, Gal, D'Antonio, Giovanni, Goldshtein, Sasha, Das, Dipanjan·2025·arXiv
DOI 10.48550/arxiv.2509.07968arXiv 2509.07968
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongside improvements in the autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5. This work provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations. The benchmark dataset, evaluation code, and leaderboard are available at: https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.

AI snapshot

Written from the paper’s full text.

TL;DR

SimpleQA Verified is a 1,000-prompt short-form factuality benchmark curated from OpenAI's SimpleQA to reduce noisy labels, topical bias, and redundancy. Gemini 2.5 Pro achieves the best reported F1-score of 55.6, outperforming GPT-5 and other frontier models.

Key findings

  • The benchmark was reduced from 4,326 SimpleQA questions to 1,000 through filtering steps including unique source documents, semantic and TF-IDF de-duplication, publisher controls, topic/answer-type balancing, source reconciliation, and headroom selection, removing 76.9% of samples overall.
  • Gemini 2.5 Pro leads with an F1-score of 55.6, followed by GPT-5 at 52.3 and o3 at 51.9; Claude Opus 4 scores 28.3 and Claude Sonnet 4 scores 18.7.
  • Autorater improvements include rewriting numeric ground truths with acceptable ranges, clarifying direct-answer and hedging rules, and adding punting examples; the prompt grows from 963 to 1,124 words (+15%).
  • Metadata enrichment classifies 3.7% of questions as requiring reasoning and 7.3% as multi-step.

Methods

  • Multi-stage curation of SimpleQA: ensuring unique source documents, semantic de-duplication with Gemini Embeddings at cosine cutoff 0.77, TF-IDF de-duplication at cosine threshold 0.4, respecting web publisher robots.txt controls, balancing answer types and topics, reconciling conflicting sources, and selecting the 1,000 most difficult remaining questions.
  • Evaluation uses the same grading scheme and metrics as Wei et al. (2024a): responses graded correct, incorrect, or not attempted, with final F1-score as the harmonic mean of overall accuracy and accuracy given attempted.
  • Models are evaluated with standard API parameters and without tools; gpt-4.1-2025-04-14 is used as the autorater.

Limitations

  • The paper does not state limitations of SimpleQA Verified itself; it states that enabling tools results in near-perfect performance, so the benchmark should be employed for measuring parametric factuality only.
  • The paper notes that cleaning removes erroneous and ambiguous questions, making the benchmark easier, which is balanced by adversarially selecting challenging samples in Section 2.6.
  • Evident from the text: SimpleQA Verified is derived from SimpleQA, is limited to 1,000 short-form prompts, and relies on an LLM autorater, so it may not capture long-form factuality, retrieval-augmented settings, or broader factual domains beyond its curated distribution.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2509.07968

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

1 of 20 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Findingcitation
    § 1. Introduction · p. 1

    SimpleQA questions are derived from a narrow distribution of source documents due to substantial human rater biases.

    “benchmark questions are derived from a narrow distribution of source documents due to substantial human rater biases”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingnone given
    § SimpleQA Verified : A Reliable Factuality Benchmark to Measure Parametric Knowledge · p. 1

    SimpleQA Verified is a 1,000-prompt benchmark for evaluating Large Language Model short-form factuality based on OpenAI's SimpleQA.

    “We introduce SimpleQA Verified , a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA.”
    Quote in the paperSubstantial claim
  • Findingcitation
    § 1. Introduction · p. 1

    SimpleQA suffers from incorrect ground truths.

    “Additionally, SimpleQA suffers from incorrect ground truths and disproportionally leans toward specific topics and question formats.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation
    § 1. Introduction · p. 1

    SimpleQA contains a high degree of redundancy with semantically similar or lexically overlapping questions.

    “The dataset contains a high degree of redundancy with semantically similar or lexically overlapping questions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § 2.4. Ensuring Diversity Across Answer Types and Topics · p. 5

    Sub-sampling to balance answer types and question topics reduced the dataset from 1,855 to 1,218 (-34.3%).

    “As part of the described process, our dataset shrinks from 1,855 to 1,218 ( -34 . 3%) examples.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three models used for difficulty filtering (GPT-4o, Gemini 2.0 Flash, and Claude 3.7 Sonnet) are representative of leading frontier models for identifying the hardest questions.
  • Classified as probably not a substantial claim. 8
  • May combine two claims. 18
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 2.7. Manual Review & Metadata Enrichment · p. 6

    3.7% of questions in SimpleQA Verified are classified as requiring reasoning and 7.3% are multi-step.

    “3 . 7% of questions are classified as requiring reasoning and 7 . 3% are multi-step.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Geva et al., 2021 source
    • cites Yang et al., 2018 source

      Section 4 (Dataset Analysis; page unavailable) reports reasoning categories and percentages for HotpotQA, not SimpleQA Verified. Searches for “3.7%,” “7.3%,” and “SimpleQA Verified” found no passage in our copy supporting the attributed figures. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § SimpleQA Verified : A Reliable Factuality Benchmark to Measure Parametric Knowledge · p. 1

    On SimpleQA Verified, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5.

    “On this new benchmark, Gemini 2.5 Pro achieves a state-ofthe-art F1-score of 55.6, outperforming other frontier models, including GPT-5.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation
    § 5. Conclusion · p. 10

    SimpleQA Verified is a rigorously curated 1,000-prompt benchmark designed to address the limitations of SimpleQA, including human rater and topical biases, incorrect labels, and question redundancy.

    “This paper presents SimpleQA Verified , a rigorously curated 1,000-prompt benchmark designed to address the limitations of its predecessor, SimpleQA (Wei et al., 2024a), including human rater and topical biases, incorrect labels, and question redundancy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al. (2024a)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingcitation
    § 2.4. Ensuring Diversity Across Answer Types and Topics · p. 4

    In the original SimpleQA, 32.8% of questions require a date and 24.1% require a person's name as the answer.

    “For instance, 32.8% of questions in SimpleQA require a date and 24.1% require a person's name as the answer (see Figure 1).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 3.1. Methodology · p. 7

    The existing prompted autorater exhibits high uncertainty for certain recurring patterns of model responses, which highlight possible headroom for improvements in the grading process.

    “We find several recurring patterns in models responses that frequently confuse the autorater with respect to the assigned grade.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.5. Reconciliation of Conflicting Sources · p. 6

    Reconciliation of conflicting sources for numeric answer types reduced the dataset from 1,117 to 1,073 (-3.9%).

    “This lowers the dataset size from 1,117 to 1,073 ( -3 . 9%).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 2.2. Removing Highly Similar Questions · p. 3

    Semantic de-duplication using Gemini Embeddings with a cosine similarity cutoff of 0.77 reduced the number of samples from 3,095 to 2,871 (-7.2%).

    “De-duplicating the dataset semantically using embeddings reduced the number of samples in our dataset from 3,095 to 2,871 ( -7 . 2%).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lee et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The three models used for difficulty filtering (GPT-4o, Gemini 2.0 Flash, and Claude 3.7 Sonnet) are representative of leading frontier models for identifying the hardest questions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.6. Increasing Benchmark Headroom · p. 6

    Filtering the 1,000 most difficult questions from the remaining samples reduced the dataset size from 1,073 to 1,000 (-6.8%).

    “This brings down the dataset size from 1,073 to 1,000 ( -6 . 8%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three models used for difficulty filtering (GPT-4o, Gemini 2.0 Flash, and Claude 3.7 Sonnet) are representative of leading frontier models for identifying the hardest questions.

  • May combine two claims.

    Findingown data
    § 4. Results · p. 9

    Gemini 2.5 Pro leads SimpleQA and SimpleQA Verified in both Accuracy and Accuracy Given Attempted (Acc.|Attempted) metrics, resulting in the highest F1-Score among frontier models.

    “Gemini 2.5 Pro leads SimpleQA and SimpleQA Verified in both Accuracy and Accuracy Given Attempted (Acc.|Attempted) metrics, resulting in the highest F1-Score among frontier models.”
    • Also stated in: § 5. Conclusion · p. 10 · boosted
      “On this more challenging and reliable evaluation set, Gemini 2.5 Pro establishes a new state-of-the-art, highlighting the benchmark's ability to differentiate frontier model capabilities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2. Removing Highly Similar Questions · p. 3

    SimpleQA contains 119 questions (2.7% of dataset) asking about the founding dates of different Colombian municipalities.

    “for example, SimpleQA contains 119 questions (2.7% of dataset) asking about the founding dates of different Colombian municipalities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4. Results · p. 9

    GPT 4o, Claude Opus 4, and Claude Sonnet 4 score statistically significantly worse on SimpleQA Verified compared to SimpleQA.

    “The results of GPT 4o , Claude Opus 4 , Claude Sonnet 4 are statistically significantly worse on SimpleQA Verified compared to SimpleQA , whereas o4-mini 's score improves.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

  • May combine two claims.

    Findingown data
    § 1. Introduction · p. 2

    Gemini 2.5 Pro leads performance on both the original SimpleQA and SimpleQA Verified benchmarks, outperforming GPT-5 and Claude Opus 4.

    “Our results on a suite of leading models show that Gemini 2.5 Pro (Gemini Team, Google, 2025) leads performance on both the original SimpleQA as well as SimpleQA Verified benchmarks, outperforming even more recently released frontier models including GPT-5 (OpenAI, 2025b) and Claude Opus 4 (Anthropic, 2025a).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Gemini Team, Google, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites OpenAI, 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Anthropic, 2025a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2. Removing Highly Similar Questions · p. 4

    TF-IDF de-duplication with a cosine similarity threshold of 0.4 reduced the dataset size from 2,871 to 2,664 (-7.2%).

    “This further reduces our dataset size from 2,871 to 2,664 ( -7 . 2%).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jones, 1972

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § SimpleQA Verified : A Reliable Factuality Benchmark to Measure Parametric Knowledge · p. 1

    SimpleQA Verified provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations.

    “This work provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § SimpleQA Verified : A Reliable Factuality Benchmark to Measure Parametric Knowledge · p. 1

    SimpleQA Verified addresses critical limitations in OpenAI's SimpleQA, including noisy and incorrect labels, topical biases, and question redundancy.

    “It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § SimpleQA Verified : A Reliable Factuality Benchmark to Measure Parametric Knowledge · p. 1

    SimpleQA Verified was created through a multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation, alongside improvements in the autorater prompt.

    “SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongside improvements in the autorater prompt.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § 2. SimpleQA Verified · p. 2

    The issues in the original SimpleQA benchmark were likely caused by misaligned incentives of human raters who created the original benchmark prompts and missing subsequent filtering and validation stages.

    “likely caused by misaligned incentives of human raters who created the original benchmark prompts and and missing subsequent filtering and validation stages”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 1. Introduction · p. 2

    By applying this rigorous methodology and further improving the autorater prompt, the authors provide a more reliable benchmark for measuring parametric factuality, designed to be evaluated without any tools.

    “By applying this rigorous methodology and further improving the autorater prompt, we provide a more reliable benchmark for measuring parametric factuality, designed to be evaluated without any tools (i.e. search).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 3.2. Autorater Improvements · p. 9

    The new guidelines ensure that the strategy of iterating various possible guesses is consistently judged as not attempted and thus is not beneficial.

    “This is important to ensure that the strategy of iterating various possible guesses (one of which may be correct) is consistently judged as not attempted and thus is not beneficial.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § 3.2. Autorater Improvements · p. 9

    Diversified examples of punting styles in the few-shot examples ensure that such predicted answers are consistently graded as not attempted.

    “In the few-shot examples included in the prompt we incorporate demonstrations of additional punting styles, to ensure that such predicted answers are consistently graded as not attempted .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 3.1. Methodology · p. 7

    Modern AI systems often produce longer and more elaborate responses that contain more information than necessary to strictly answer the provided question.

    “Despite the focused nature of the questions in SimpleQA Verified , modern AI systems still often produce longer and more elaborate responses. These typically contain more information than is necessary to strictly answer the provided question.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4. Results · p. 9

    The cleaning process removes erroneous and ambiguous questions, making the benchmark easier, and this is balanced by adversarially selecting a subset of challenging samples.

    “our cleaning process described in Section 2 removes erroneous and ambiguous questions which makes the benchmark easier. This is balanced by adversarially selecting a subset of challenging samples in Section 2.6.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § 1. Introduction · p. 2

    SimpleQA Verified was curated through a multi-stage process involving removing duplicate sources, semantic and TF-IDF de-duplication, re-balancing of topic and answer-type distributions, reconciliation of conflicting sources, and filtering reference URLs to align with web publishers' crawling preferences.

    “SimpleQA Verified was curated through a multi-stage process involving removing duplicate sources, semantic and TF-IDF de-duplication, re-balancing of topic and answer-type distributions to ensure question diversity, reconciliation of conflicting sources to verify ground truths, and a filtering step to align reference URLs in our benchmark with the crawling preferences of their web publishers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5. Conclusion · p. 10

    SimpleQA Verified is a more challenging and reliable evaluation set.

    “On this more challenging and reliable evaluation set, Gemini 2.5 Pro establishes a new state-of-the-art, highlighting the benchmark's ability to differentiate frontier model capabilities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

  • May combine two claims.

    Interpretationreasoning
    § 2.2. Removing Highly Similar Questions · p. 3

    SimpleQA contains many highly similar questions which often seem to stem from the same human rater, likely a result of misaligned rating incentives.

    “SimpleQA (Wei et al., 2024a) contains many highly similar questions which often seem to stem from the same human rater (no rater IDs were provided in OpenAI's release), likely a result of misaligned rating incentives.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 2.4. Ensuring Diversity Across Answer Types and Topics · p. 5

    The set of 1,218 questions is sizeable enough to allow for meaningful statistics and ensures SimpleQA Verified is a well-balanced factuality benchmark with diverse questions.

    “Using our filtering technique, we obtain a set of 1,218 questions which is still sizeable enough to allow for meaningful statistics, while also ensuring SimpleQA Verified is a well-balanced factuality benchmark with diverse questions.”
    Quote in the paperSubstantial claim
26 of 26

  • The paper relies on this without support.

    The three models used for difficulty filtering (GPT-4o, Gemini 2.0 Flash, and Claude 3.7 Sonnet) are representative of leading frontier models for identifying the hardest questions.

    “we randomly sampled a question which all of GPT-4o (OpenAI, 2024a), Gemini 2.0 Flash (Pichai et al., 2024), and Claude 3.7 Sonnet (Anthropic, 2025b) answered incorrectly”
    “discarding questions which all leading frontier models solve correctly”

  • The paper relies on this without support.

    The autorater gpt-4.1-2025-04-14 accurately and reliably grades answers on SimpleQA Verified.

    “In our results, we report the same metrics computed in Wei et al. (2024a) and use gpt-4.1-2025-04-14 as an autorater.”
2 of 2

Cites

Flagged first, then by citations

4 of the 25 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2509.07968
arXiv
2509.07968
OpenAlex
W4416068860
Semantic Scholar
f299355b92f9d8ad9471a2ffb8071748952934c1
Cited by
1
References
25
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, hallucination mitigation, label noise, Gemini 2.5 Pro

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review