Measuring short-form factuality in large language models
Open accessAbstract
We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.
Verified 2026-10-22 · 1 reference not checked yet
AI snapshot
Written from the paper’s full text.
TL;DR
SimpleQA is a benchmark of 4,326 short, fact-seeking questions with single indisputable answers, designed to be challenging for frontier models and easy to grade. It measures short-form factuality and calibration, finding that evaluated frontier models score below 50% correct and tend to overstate confidence.
Key findings
- SimpleQA contains 4,326 questions; the most common topics were Science & Technology (n=858), Politics (n=709), and Art (n=550).
- Frontier models scored below 50% correct: GPT-4o had 38.2 correct and F-score 38.4; OpenAI o1-preview had 42.7 correct and F-score 44.8; Claude-3.5 Sonnet had 28.9 correct and F-score 35.0; Claude-3-opus had 23.5 correct and F-score 29.3.
- Calibration analyses showed models consistently overstate stated confidence; o1-preview was more calibrated than o1-mini, and GPT-4o was more calibrated than GPT-4o-mini.
- A third trainer answered 1,000 randomly selected examples with 94.4% performance by the prompted ChatGPT grader; after manual review, the authors estimate the benchmark error rate is around 3%, assuming no false positives from the grader.
Methods
- Two-stage data collection: AI trainers created question-answer pairs with supporting evidence links, then a second AI trainer independently answered each question without seeing the original answer; only matching answers were kept.
- Questions had to have a single indisputable answer, not change over time, be answerable as of December 31, 2023, and be challenging: trainers reviewed four OpenAI model completions and required at least one incorrect completion, mostly using GPT-4 variants and later one GPT-3.5.
- Grading used a prompted ChatGPT classifier to label responses as correct, incorrect, or not attempted; metrics included overall correct, correct given attempted, and F-score as their harmonic mean, with calibration measured via stated confidence and 100 repeated samples at temperature 1.
Limitations
- The paper states that SimpleQA only measures factuality under the constrained setting of short, fact-seeking queries with a single verifiable answer; whether short-form factuality correlates with long-form factuality remains an open research question.
- The paper states that F-score has a loophole: if model performance is below 50%, it always makes sense for the model to guess if it is at least 50% sure it will be correct.
- The paper states it did not do a formal study of grader performance, though a manual check of 100 correct, 100 incorrect, and 100 not attempted completions found only two disagreements; the estimated benchmark error rate of around 3% assumes no false positives from the prompted ChatGPT grader.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
1 of 3 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Findingown data§ 4 Measuring calibration · p. 7
On SimpleQA, models consistently overstate their confidence (stated confidence exceeds actual accuracy).
“However, the fact that performance is well below the line y = x means that models consistently overstate their confidence.”
Quote in the paperSubstantial claim- Rests on an assumption: The model's stated confidence percentage reflects its confidence.
- Findingown data§ 2.2 Data quality · p. 4
The remaining 2.8% of errors from the third trainer revealed real issues with the data.
“The remaining 2.8% of the errors from the third trainer revealed real issues with the data.”
Quote in the paperSubstantial claim - Findingown data§ 2.2 Data quality · p. 4
In a random sample of 1,000 SimpleQA examples, the third trainer's answers were graded 94.4% correct by the prompted ChatGPT grader.
“After the dataset was finalized with reference answers agreed on by two trainers, we did an additional quality check by randomly selecting 1,000 examples and having them answered by a third trainer. According to the prompted ChatGPT grader, the performance of the third trainer was 94.4%.”
Quote in the paperSubstantial claim - Findingown data§ 2.2 Data quality · p. 4
SimpleQA's error rate is estimated around 3%, assuming no false positives from the prompted ChatGPT grader.
“Hence, we estimate that the error rate of our benchmark is around 3%, assuming no false positives from the prompted ChatGPT grader.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The prompted ChatGPT grader produced no false positives in the 1,000-example quality check.
- Findingown data§ 1 Introduction · p. 2
SimpleQA contains 4,326 short, fact-seeking questions.
“We present a benchmark called SimpleQA , which contains 4,326 short, fact-seeking questions.”
Quote in the paperSubstantial claim
- Findingown data§ 2.4 Grading and metrics · p. 6
None of the models evaluated in this paper meet the bar of getting at least 90% of attempted problems correct.
“At p = 9, the weighted sum of the scores for a model would only be positive if the model was getting at least 90% of the problems it attempted correct (a bar that none of the models we evaluate in this paper currently meet).”
Quote in the paperNot a substantial claim - Findingown data§ 4 Measuring calibration · p. 7
Across all models, accuracy increases with frequency of the most-frequent answer.
“We see across all models that accuracy increases with frequency, and that o1-preview has the highest level of calibration, where the frequency of the response is roughly equivalent to the accuracy of the response.”
Quote in the paperSubstantial claim- Rests on an assumption: Higher frequency of an answer typically indicates that the model is more confident in that answer.
- Findingown data§ 2.2 Data quality · p. 4
Of the 41 actually incorrect answers, six were due to the third trainer giving an answer that contradicted the source they cited.
“Of the remaining 41 (4.1%) actually incorrect answers, we found that seven were due to the third trainer not answering the question fully (e.g., giving only the year when the question asks for month and year), and six were due to the third trainer giving an answer that contradicted the source they cited (e.g., they misread the own source they cited).”
Quote in the paperNot a substantial claim - Findingown data§ 2.2 Data quality · p. 4
Of the 56 answers classified incorrect by the prompted ChatGPT grader, 15 were false negatives.
“Of the 5.6% of answers (56/1000) classified as incorrect by the prompted ChatGPT grader, we did a manual examination of all examples and found that fifteen were false negatives from the autograder.”
Quote in the paperSubstantial claim - Findingown data§ 2.2 Data quality · p. 4
Of the 41 actually incorrect answers, seven were due to the third trainer not answering the question fully.
“Of the remaining 41 (4.1%) actually incorrect answers, we found that seven were due to the third trainer not answering the question fully (e.g., giving only the year when the question asks for month and year), and six were due to the third trainer giving an answer that contradicted the source they cited (e.g., they misread the own source they cited).”
Quote in the paperNot a substantial claim - Interpretationreasoning§ Abstract · p. 1
SimpleQA is challenging.
“First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3 Evaluation of models · p. 6
SimpleQA is likely a challenging dataset for frontier models generally.
“We see that Claude's performance is also not super high, so it is likely that SimpleQA is a challenging dataset for frontier models generally.”
Quote in the paperSubstantial claim - Study design and conductnone given§ Abstract · p. 1
SimpleQA is adversarially collected against GPT-4 responses.
“First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses.”
Quote in the paperSubstantial claim - Interpretationown data§ 1 Introduction · p. 2
SimpleQA is created to be challenging for frontier models.
“SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
Quote in the paperSubstantial claim - Interpretationcitation§ 2.4 Grading and metrics · p. 6
For F-score, if model performance is below 50%, it always makes sense for the model to try to guess if it is at least 50% sure that it will get an answer correct.
“However, an issue with F-score is that if model performance is below 50%, it always makes sense for the model to try to guess if it is at least 50% sure that it will get an answer correct (see Appendix B for why this is the case; Kalai (2024)).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kalai (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data§ 4 Measuring calibration · p. 7
There is a lot of room to improve the calibration of large language models in terms of stated confidence.
“Hence, there is a lot of room to improve the calibration of large language models in terms of stated confidence.”
Quote in the paperSubstantial claim- Rests on an assumption: The model's stated confidence percentage reflects its confidence.
- Study design and conductreasoning§ Abstract · p. 1
Responses in SimpleQA are easy to grade because questions are created such that there exists only a single, indisputable answer.
“Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer.”
Quote in the paperSubstantial claim - Study design and conductown data§ 2.4 Grading and metrics · p. 5
In a manual review of 100 correct, 100 incorrect, and 100 not attempted completions, there were only two disagreements with the prompted grader.
“Of 100 correct, 100 incorrect, and 100 not attempted completions we manually read, we only found two disagreements with the prompted grader.”
Quote in the paperSubstantial claim - Interpretationreasoning§ B Guessing strategy and F-score · p. 13
If a model has a greater than 1/2 chance of being correct, its expected F-score from guessing is better than the score from not guessing, regardless of the specific values for c, i, and n.
“If you have a greater than 1 / 2 chance of being correct, your expected score from guessing is better than the score from not guessing, regardless of the specific values for c , i , and n .”
Quote in the paperSubstantial claim - Interpretationnone given§ 5 Related work and discussion · p. 8
SimpleQA only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer.
“A main limitation with SimpleQA is that while it is accurate, it only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer.”
Quote in the paperSubstantial claim - Interpretationreasoning§ B Guessing strategy and F-score · p. 13
Since the denominators are adjusted similarly whether the guess is correct or incorrect, guessing with a probability greater than 1/2 yields a better score.
“Since the denominators are adjusted similarly whether the guess is correct or incorrect, guessing with a probability > 1 / 2 yields a better score.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 25
- May combine two claims. 12
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5 Related work and discussion · p. 8Kadavath et al. (2022) found that language models were increasingly calibrated as a function of training compute.
“Notably, Kadavath et al. (2022) studied calibration in language models, finding that they were increasingly calibrated as a function of training compute, where calibration is measured by the having the language model give the probability of a true statement.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kadavath et al. (2022) source
Section 2 (p. 8) reports that calibration generally improves with model size and capability, not specifically as a function of training compute. The Glossary (p. 5) defines P(True) as the probability that a particular answer is correct, but the paper also measures calibration by comparing predicted probabilities with frequencies of correct predictions. The citing sentence therefore captures the broad finding but misstates the scaling variable and narrows the calibration measure. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingreasoning§ 1 Introduction · p. 2SimpleQA is fast and simple to run, as questions and answers are very short.
“SimpleQA is fast and simple to run, as questions and answers are very short.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3 Evaluation of models · p. 6Claude models tend to not attempt questions more often than GPT-4o models on SimpleQA.
“Another interesting observation with the Claude models is that they tend to not attempt questions more often than the GPT-4o models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ 2.3 Dataset diversity · p. 4In SimpleQA, 32.8% of answers were dates, 24.1% were a person, 15.3% were a number, 9.9% were a place, and 18.0% were classified as 'other.'
“Using ChatGPT to classify types of answers, we found that 32.8% of answers were dates, 24.1% of answers were a person, 15.3% answers were a number, 9.9% answers were a place, and 18.0% answers were classified as 'other.'”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ 2.3 Dataset diversity · p. 4SimpleQA's most common topics are Science & Technology (n=858), Politics (n=709), and Art (n=550).
“The most common topics were Science & Technology (n=858), Politics (n=709), and Art (n=550).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4 Measuring calibration · p. 7In frequency-based calibration, o1-preview is more calibrated than o1-mini, and GPT-4o is more calibrated than GPT-4o-mini.
“Similar to calibration via stated confidence plot above, we again see o1-preview is more calibrated than o1-mini, and gpt4o is more calibrated than gpt4o-mini.”
Quote in the paperSubstantial claim- Rests on an assumption: Higher frequency of an answer typically indicates that the model is more confident in that answer.
May combine two claims.
Findingown data§ 4 Measuring calibration · p. 7On SimpleQA calibration via stated confidence, o1-preview is more calibrated than o1-mini, and GPT-4o is more calibrated than GPT-4o-mini.
“We see that o1-preview is more calibrated than o1-mini, and gpt4o is more calibrated than gpt4o-mini, which is consistent with prior work showing that larger models are more calibrated.”
Quote in the paperSubstantial claim- Rests on an assumption: The model's stated confidence percentage reflects its confidence.
May combine two claims.
Findingnone given§ 1 Introduction · p. 2GPT-4o and Claude both score less than 50% on SimpleQA.
“SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ 1 Introduction · p. 2SimpleQA contains questions from a wide range of topics, including history, science & technology, art, geography, TV shows, etc.
“SimpleQA contains questions from a wide range of topics, including history, science & technology, art, geography, TV shows, etc.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingown data§ Abstract · p. 1SimpleQA is a benchmark that evaluates the ability of language models to answer short, fact-seeking questions.
“We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 2.3 Dataset diversity · p. 4wikipedia.com is by far the biggest source for SimpleQA, being one of the sources for 3.5k of 4.3k questions, followed by fandom.com (410 questions), ac.uk (154 questions), and imdb.com (121 questions).
“As for diversity in sources, we see that wikipedia.com is by far the biggest source (one of the sources for 3.5k of 4.3k questions), followed by fandom.com (410 questions), ac.uk (154 questions), and imdb.com (121 questions).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3 Evaluation of models · p. 6Claude-3.5 Sonnet has much fewer correct questions than GPT-4o, but also attempts much fewer questions, resulting in a similar F-score.
“For instance, Claude-3.5 Sonnet has much fewer correct questions than GPT-4o, but also attempts much fewer questions, resulting in a similar F-score.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3 Evaluation of models · p. 6On SimpleQA, larger models have higher performance than smaller models; specifically GPT-4o outperforms GPT-4o-mini, o1-preview outperforms o1-mini, and Claude-3-opus has the highest performance of the Claude-3 series.
“As expected, we see that larger models have higher performance than smaller models (GPT-4o outperforms GPT-4o-mini; o1-preview outperforms o1-mini and opus is the highest performance of the claude-3 series).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 2.2 Data quality · p. 4The most common issues in SimpleQA are ambiguous questions, reputable sources giving contradictory information, and having more than one correct answer.
“The most common issues were ambiguous questions (e.g., not specifying driving vs flying distance between cities), reputable sources giving contradictory information (e.g., history.com and wikipedia.com give different dates for when Nixon retired from the US Naval Reserve), and having more than one correct answer (e.g., John Lennon's psychedelic Rolls-Royce was shown at the Pacific National Exhibi”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4 Measuring calibration · p. 7o1-preview has the highest level of calibration among evaluated models, with response frequency roughly equivalent to accuracy.
“We see across all models that accuracy increases with frequency, and that o1-preview has the highest level of calibration, where the frequency of the response is roughly equivalent to the accuracy of the response.”
Quote in the paperSubstantial claim- Rests on an assumption: Higher frequency of an answer typically indicates that the model is more confident in that answer.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 2 Data collection and verification · p. 3AI trainers created question and answer pairs for SimpleQA.
“First, AI trainers (i.e., human annotators) created question and answer pairs.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ 2.2 Data quality · p. 4Questions were kept only if there were two unique website domain names among the 2-4 sources found by the two AI trainers.
“To improve the correctness of reference answers, we also only kept questions for which there were two unique website domain names among the 2-4 sources found by the two AI trainers (one source from the first AI trainer, and 1-3 from the second AI trainer).”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone given§ 2.4 Grading and metrics · p. 5SimpleQA completions are graded by a prompted ChatGPT classifier that sees both the predicted answer and the reference answer and grades responses as correct, incorrect, or not attempted.
“To grade completions, we use a prompted ChatGPT classifier that sees both the predicted answer and the reference answer, and grades responses as either 'correct,' 'incorrect,' or 'not attempted.'”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductown data§ 2 Data collection and verification · p. 3Questions in SimpleQA were independently answered by another AI trainer and only kept if answers from both trainers matched.
“Then, questions were independently answered by another AI trainer and only kept if answers from both trainers matched.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationcitation§ 5 Related work and discussion · p. 8The finding that frequency of answer correlates with accuracy is consistent with Wang et al. (2023).
“Our finding that frequency of answer correlates with accuracy is also consistent with Wang et al. (2023), which found a similar result for a different model on a math word problem benchmark.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wang et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: Higher frequency of an answer typically indicates that the model is more confident in that answer.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 2.4 Grading and metrics · p. 6At p = 9, the weighted sum of scores for a model would only be positive if the model was getting at least 90% of the problems it attempted correct.
“At p = 9, the weighted sum of the scores for a model would only be positive if the model was getting at least 90% of the problems it attempted correct (a bar that none of the models we evaluate in this paper currently meet).”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ 2.1 Question and answer criteria · p. 3Questions were required to be answerable as of December 31, 2023.
“Finally, we required questions to be answerable as of December 31, 2023, so that we could equally evaluate all models trained with data knowledge cutoffs up to that date.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ 2.1 Question and answer criteria · p. 3Questions were kept only if at least one of four model completions was incorrect.
“At least one of the four completions must be incorrect for the trainer to continue with that question; otherwise, the trainer was instructed to create a new question.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone given§ Abstract · p. 1Each answer in SimpleQA is graded as either correct, incorrect, or not attempted.
“Each answer in SimpleQA is graded as either correct, incorrect, or not attempted.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoning§ 2.1 Question and answer criteria · p. 3Each reference answer in SimpleQA is supported by a link to a webpage.
“When AI trainers initially created a question and reference answer, they were also asked to provide a link to the webpage that supports the reference answer to the question.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ 2.1 Question and answer criteria · p. 3SimpleQA questions have a single, indisputable answer.
“Here we only focus on objective knowledge and force questions to be written such that they only have a single, indisputable answer.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoning§ 2.1 Question and answer criteria · p. 3SimpleQA questions are written so that their answers do not change over time.
“To keep this dataset evergreen, questions are written so that their answers would not change over time, which can require increasing the degree of specificity.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown data§ 2.2 Data quality · p. 4Questions were removed if answers from two trainers didn't agree or if the second trainer said the question wasn't timeless or didn't have an indisputable answer.
“After the verification stage, we removed questions from the dataset if answers from two trainers didn't agree according to a prompted ChatGPT classifier, or if the second trainer said the question wasn't timeless or didn't have an indisputable answer.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown data§ 4 Measuring calibration · p. 7The positive correlation between stated confidence and accuracy is a reassuring sign that models have some notion of confidence.
“The positive correlation between stated confidence and accuracy is a reassuring sign that models have some notion of confidence.”
Quote in the paperNot a substantial claim- Rests on an assumption: The model's stated confidence percentage reflects its confidence.
Classified as probably not a substantial claim.
Study design and conductnone given§ 2.2 Data quality · p. 4Each question was answered by an independent AI trainer without access to the answer given by the question creator.
“In the verification stage of this dataset, each question was answered by an independent AI trainer without access to the answer given by the question creator.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 2.2 Data quality · p. 3ChatGPT classifiers were used to detect criteria violations during question creation.
“First, during the question creation stage, we ran a series of few-shot-prompted ChatGPT classifiers to detect criteria violations such as not specifying units, having answers that change over time, or having multiple answers.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ 1 Introduction · p. 2With 4,326 questions in the dataset, SimpleQA should have relatively low run-to-run variance.
“Additionally, with 4,326 questions in the dataset, SimpleQA should have relatively low run-to-run variance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 2.4 Grading and metrics · p. 6For F-score, a model with correct-given-attempted of 80% would only need overall-correct of 19% to get an F-score of 30%.
“a model that has correct-given-attempted of 80% would only need to get overall-correct of 19% to get an F-score of 30%.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Current frontier models sometimes produce false outputs or answers that are not substantiated by evidence (hallucinations).
“Current frontier models sometimes produce false outputs or answers that are not substantiated by evidence, a problem known as 'hallucinations.'”
The paper relies on this without support.
Such hallucinations are one of the major barriers for broader adoption of general artificial intelligence like large language models.
“Such hallucinations are one of the major barriers for broader adoption of general forms artificial intelligence like large language models.”
The paper relies on this without support.
TriviaQA and Natural Questions are too easy for today's language models.
“which were good datasets at the time but are too easy for today's language models.”
The authors state this as an assumption.
The prompted ChatGPT grader produced no false positives in the 1,000-example quality check.
“assuming no false positives from the prompted ChatGPT grader.”
The paper relies on this without support.
Higher frequency of an answer typically indicates that the model is more confident in that answer.
“Higher frequency typically indicates that the model is more confident in its answers, as the model is giving the same answer repeatedly.”
The paper relies on this without support.
The model's stated confidence percentage reflects its confidence.
“One way to measure calibration is to directly ask the language model to state its confidence in its answer using a prompt like: 'Please give your best guess, along with your confidence as a percentage that that is the correct answer'”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BrowseComp instructions largely followed SimpleQA, with human trainers creating challenging, fact-seeking questions with single, indisputable, short answers that would not change over time and were supported by evidence.
“The instructions of BrowseComp largely followed those of SimpleQA (Wei et al., 2024)-the human trainers created challenging, factseeking questions with single, indisputable, short answers that would not change over time and were supported by evidence.”
From BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025) · cites this paper as Wei et al., 2024
Models are evaluated using a GPT-4O MINI auto-rater adapted from Wei et al. (2024).
“Models are evaluated using a GPT-4O MINI auto-rater adapted from Wei et al. (2024)”
From SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models (2025) · cites this paper as Wei et al. (2024)
Prior to recent benchmarks specifically designed for evaluating information retrieval capabilities, search-based agents were typically evaluated using conventional capability benchmarks, including SimpleQA and HLE.
“Prior to recent benchmarks specifically designed for evaluating information retrieval capabilities [1, 5, 8, 14, 17, 19 ? ], search-based agents were typically evaluated using conventional capability benchmarks, including SimpleQA [16] and Humanity's Last Exam (HLE) [12].”
From Search-Time Data Contamination (2025) · cites this paper as [16]
SimpleQA acknowledges that all questions are solvable with knowledge existing before December 31, 2023.
“SimpleQA acknowledges that all questions are solvable with knowledge existing before December 31, 2023 [16].”
From Search-Time Data Contamination (2025) · cites this paper as [16]
Existing question-answering benchmarks focus on short-form, easily-verifiable answers and do not reflect realistic research synthesis tasks.
“Many prior works evaluate generative research synthesis systems using existing question answering benchmarks, which do not reflect realistic research synthesis tasks and instead focus on questions with short-form, easily-verifiable answers, making them severely limited for this setting [56, 55, 31, 21, 58, 53, 19, 61, 20, 22, 16, 49, 24].”
From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [55]
Several recent works focus their evaluation on question answering (QA) and factuality benchmarks with short-form, easily verifiable answers, including SimpleQA [55], FRAMES [21], GAIA [31], BrowserComp [56], BrowserComp-Plus [8] WebWalkerQA [58], DeepResearch Arena [54] and others.
“These prior benchmarks include SimpleQA [55], FRAMES [21], GAIA [31], BrowserComp [56], BrowserComp-Plus [8] WebWalkerQA [58], DeepResearch Arena [54] and others traditionally used to evaluate retrieval-augmented generation [53, 19, 61, 20, 22, 16, 49, 24].”
From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [55]
Cites
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 2× · see citing works
“Compared to older benchmarks such as TriviaQA (Joshi et al., 2017) or Natural Questions (Kwiatkowski et al., 2019) that are now saturated, SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2× · see citing works
“Compared to older benchmarks such as TriviaQA (Joshi et al., 2017) or Natural Questions (Kwiatkowski et al., 2019) that are now saturated, SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
BackgroundTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 1× · see citing works
“…benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
BackgroundSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Self-citationHale, E. J. (2000) · Nursing · doi:10.1097/00152193-200030090-00033 · Cited by 446
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
FELM: Benchmarking Factuality Evaluation of Large Language ModelsPreprint
Chen, S., Zhao, Y., Zhang, J., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.00741 · Cited by 89 · cited 1× · see citing works
“…benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
BackgroundTeaching Models to Express Their Uncertainty in Words
Lin, S., Hilton, J., & Evans, O. (2022) · arXiv · doi:10.48550/arxiv.2205.14334 · Cited by 59 · cited 1× · see citing works
“Other work has attempted to improve the calibration of language models (Lin et al., 2022b; Agrawal et al., 2024), which could lead to increased adoption of language models in real-world settings.”
BackgroundDo Language Models Know When They’re Hallucinating References?
Agrawal, A., Suzgun, M., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2305.18248 · Cited by 37 · cited 1× · see citing works
“Other work has attempted to improve the calibration of language models (Lin et al., 2022b; Agrawal et al., 2024), which could lead to increased adoption of language models in real-world settings.”
BackgroundLong-form factuality in large language models
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Huang, J., Tran, D., Peng, D., Liu, R., Huang, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.18802 · Cited by 36 · cited 2× · see citing works
“Other recent and related benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
BackgroundSelf-citationFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · cited 1× · see citing works
“…benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
BackgroundEvaluating Hallucinations in Chinese Large Language ModelsPreprint
Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.03368 · Cited by 9 · cited 2× · see citing works
“Our finding that frequency of answer correlates with accuracy is also consistent with Wang et al. (2023), which found a similar result for a different model on a math word problem benchmark.”
BackgroundResultFreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7 · see citing works
Self-citationFact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · cited 1× · see citing works
“…benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
Background
Flagged first, then by citations
3 of the 19 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective
Lavrinovics, E., Biswas, R., Bjerva, J., & Hose, K. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100844 · Cited by 111 · see citing works
AI hallucinations can’t be stopped — but these techniques can limit their damage
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00068-5 · Cited by 91 · see citing works
Deliberative Alignment: Reasoning Enables Safer Language Models
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15159 · Cited by 106 · see citing works
Self-citationLarge language models for clinical decision support in gastroenterology and hepatology
Wiest, I. C., Bhat, M., Clusmann, J., Schneider, C. V., Jiang, X., & Kather, J. N. (2025) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-025-01108-1 · Cited by 33 · see citing works
“Similarly, new alignment techniques and in-model uncertainty estimation, described in a study published in Nature and in a study from OpenAI (a preprint), have shown paths to reduce the tendency of models to fabricate information 162,163 .”
Henestrosa, A. L., & Kimmerle, J. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100142 · Cited by 34 · see citing works
A benchmark of expert-level academic questions to assess AI capabilities
Center for AI Safety, Phan, L., Gatti, A., Li, N., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., et al. (2026) · Nature · doi:10.1038/s41586-025-09962-4 · Cited by 23 · see citing works
Enhancing LLM Reasoning Capabilities Through Brokered Multi-Expert Reflection
Sheokand, T., Jain, G., Bahga, A., & Madisetti, V. K. (2025) · IEEE Access · doi:10.1109/access.2025.3561693 · Cited by 7 · see citing works
“As illustrated in Figure 14, the approach employing Ministral 8B and Claude 3-haiku demonstrates a higher number of correct conclusions (69) and fewer incorrect conclusions (16), indicating that generalized experts are proficient at effectively capturing reasoning patterns.”
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., & Glaese, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12516 · Cited by 2 · see citing works
Self-citationIntroduction: (de)constructing trust in high-stakes intercultural mediation
Hu, B., Valdez, S., & Ragni, V. (2025) · Translation Studies · doi:10.1080/14781700.2025.2518577 · Cited by 1 · see citing works
Disher, T., Janoudi, G., & Rada, M. (2024) · medRxiv · doi:10.1101/2024.11.15.24317267 · Cited by 1 · see citing works
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1 · see citing works
“In the original SimpleQA (Wei et al., 2024a) paper, certain topics and answer types are overrepresented, and the distribution of answer types can skew benchmark results.”
InfluentialSearch Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
“Datasets for evaluating these LLM-IR systems span needle-in-a-haystack retrieval [29, 63], citation attribution [69, 1], and general question answering [65, 21, 60, 61].”
InfluentialReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Li, M., Zeng, Y., Cheng, Z., Ma, C., & Jia, K. (2025) · arXiv · doi:10.48550/arxiv.2508.15804 · Cited by 1 · see citing works
Continual Learning via Sparse Memory Finetuning
Lin, J., Zettlemoyer, L., Ghosh, G., Yih, W.-T., Markosyan, A., Berges, V.-P., & Oğuz, B. (2025) · arXiv · doi:10.48550/arxiv.2510.15103
“We evaluate whether methods can learn from a stream of documents by training on Active Reading-augmented (Lin et al., 2025) documents for a subset of 100 Wikipedia-grounded SimpleQA questions (Wei et al., 2022).”
Modernisierung von MINT-Praktika durch GenKI – Zugang zum forschenden Lernen?
Horst, C., Zimmermann, J., Breiling, M., et al. (2025) · Zeitschrift für Hochschulentwicklung · doi:10.21240/zfhe/sh-ki-2/04
Wang, J., Zou, X., & Huang, L. (2025) · Chinese as a Second Language (漢語教學研究—美國中文教師學會學報) The journal of the Chinese Language Teachers Association USA · doi:10.1075/csl.00069.wan
Conformal Prediction in the Age of Multimodal Foundation Models: A Survey
Stavan, C., Tilaganji, K., Krishnan, S. M., Kancheti, S. S., & Balasubramanian, V. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15120-9_13
Natural language querying of biological databases with large language models.
Makarov, V. A., Stroganov, O., Furlong, L. I., Evarts, B. M., van den Biggelaar, L., Goulas, A., Stolte, E., Marren, D., & Greiffenberg, L. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104654 · Cited by 1 · see citing works
TVM-SimpleQA: A Factuality Evaluation Dataset for Traditional Vietnamese Medicine
Nguyen, G., Nguyen, T., Nguyen, L., & Đinh, Đ. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21631-1_36
Dammu, P. P. S., Palkhiwala, A., Roosta, T. G., & Shah, C. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808627
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
“This follows current practices for building challenging benchmarks; for example, S IMPLE QA [Wei et al., 2024] was also adversarially collected against GPT-4 responses.”
Xie, S., Shi, Z., Shen, H., Wang, J., Zhong, S., Liu, M., Pan, C., Jia, P., Sun, B., Jing, X., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.21482
“This taxonomy follows the separation in recent evaluation work between factual retrieval and factuality (43), browsing or web traversal (29; 46; 10), interactive tool use (36; 50), and long-horizon agent tasks (26; 48).”
Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
Yang, C., Zhao, X., Wu, T., & Kästner, C. (2026) · arXiv · doi:10.48550/arxiv.2607.08938
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
HalluHard: A Hard Multi-Turn Hallucination Benchmark
Fan, D., Delsad, S., Flammarion, N., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2602.01031
“Given that SimpleQA’s estimated benchmark error rate is about 3% (Wei et al., 2024a), performance with web search is already near the ceiling, suggesting the benchmark is largely saturated under a browsing setting.”
Search-Time Data Contamination
Han, Z., Mankikar, M., Michael, J., & Wang, Z. (2025) · arXiv · doi:10.48550/arxiv.2508.13180
“], search-based agents were typically evaluated using conventional capability benchmarks, including SimpleQA [16] and Humanity’s Last Exam (HLE) [12].”
InfluentialPhan, L. (2025) · doi:10.21203/rs.3.rs-6615297/v1 · Cited by 65 · see citing works
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Zhu, Y., Jin, T., Pruksachatkun, Y., Zhang, A., Liu, S., Cui, S., Kapoor, S., Longpre, S., Meng, K., Weiss, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.02825 · Cited by 2 · see citing works
- Authors
- Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W.
- ORCID
- Chung, Hyung Won
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2411.04368
- OpenAlex
- W4404396608
- Semantic Scholar
- 3f99d8e6dada94f5bcfc650be3aab7a24e39bab3
- Cited by
- 14
- References
- 19
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- factuality, large language models, question answering, benchmark, confidence calibration, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
