HalluHard: A Hard Multi-Turn Hallucination Benchmark

May combine two claims.

The in-context hallucination criterion is tightly entangled with instruction-following ability and may not cleanly measure hallucination propensity.

“We argue, however, that this criterion is tightly entangled with instruction-following ability, i.e., whether the model can restrict itself to using only the given context when prompted to do so (McMillan et al., 2025), and therefore may not cleanly measure hallucination propensity.”

• cites McMillan et al., 2025 sourcePartly supported

  In §5.3 (p. 6), the paper says measured explanation faithfulness may partly reflect a model’s ability to follow instructions rather than genuine alignment between answers and explanations. That supports the concern about instruction-following as a measurement confound, but the paper does not examine an in-context hallucination criterion or whether a model restricts itself to a supplied context. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Sinha et al. (2025) cautioned this phenomenon as a self-conditioning effect.

“This phenomenon is also cautioned by Sinha et al. (2025) as self-conditioning effect.”

• cites Sinha et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Reinforcement fine-tuning can make LLMs less likely to refuse unanswerable questions.

“Song et al. (2025) warn that reinforcement fine-tuning can make LLMs less likely to refuse unanswerable questions.”

• cites Song et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Fine-tuning LLMs on human-labeled data can reduce factuality as models encounter novel knowledge or unfamiliar texts.

“Lin et al. (2024) show that fine-tuning LLMs on human-labeled data can reduce factuality as models encounter with novel knowledge or unfamiliar texts.”

• cites Lin et al. (2024) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Coding questions follow the same construction as Krishna et al. (2025) and cover Python, Scala, R, and Elixir.

“We follow the same question construction as in Krishna et al. (2025), and choose four programming languages: Python , Scala , R , and Elixir .”

• cites Krishna et al. (2025) sourceSupported

Classified as probably not a substantial claim.

Legal questions are taken from Magesh et al. (2025), selecting SCALR, Rule QA, Changes in Law, and Bar Exam types.

“We take existing verified legal questions from Magesh et al. (2025), selecting the following four question types: SCALR, Rule QA, Changes in Law and Bar Exam, as they incentivize open-ended answers.”

• cites Magesh et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

A clean test of in-parameter groundedness is whether models appropriately abstain when asked about non-existent entities or items.

“Beyond answerable questions, a clean test of in-parameter groundedness is whether models appropriately abstain when asked about non-existent entities or items (Bang et al., 2025; Kirichenko et al., 2025).”

• cites Bang et al., 2025 sourceSupported
• cites Kirichenko et al., 2025 sourcePartly supported

  Section 3.2 (p. 4) identifies false-premise questions as cases where models should abstain, and the abstention-detection prompt (p. 22) illustrates this with a question about leopards with wings. This supports testing whether models reject a premise involving something that does not exist. The paper does not establish that this is a clean test of in-parameter groundedness; its Limitations (p. 18) note that overlap between evaluation data and training data can confound abstention performance. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In-parameter hallucination is commonly evaluated using short-form factual prompts and long-form generation with atomic fact verification.

“In-parameter hallucination is commonly evaluated using short-form factual prompts (e.g., QA-style queries) (Lin et al., 2022; Wei et al., 2024a; Agrawal et al., 2024; Pandit et al., 2025), and also via longform generation where responses are broken into atomic facts that can be verified one by one (Min et al., 2023a; Manakul et al., 2023; Wei et al., 2024b).”

• cites Lin et al., 2022 sourceSupported
• cites Wei et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Min et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Kirichenko et al. (2025) report that adding reasoning reduces abstention.

“While Kirichenko et al. (2025) report that adding reasoning reduces abstention”

• cites Kirichenko et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Bang et al. (2025) found that the model has sufficient knowledge to identify non-existing entities.

“Bang et al. (2025) had a similar observation by investigating the model's abstention behavior across mixed or completely fabricated entities, and found that the model has sufficient knowledge to identify non-existing entities.”

• cites Bang et al. (2025) sourcePartly supported

  The paper’s discussion of place-frequency results (section headed “Lower values indicate better performance”; page unavailable) says models may have sufficient knowledge to identify non-existing entities associated with high-frequency places. It also says they are more uncertain about middle-frequency places. The NonExistentRefusal results (C.3.1–C.3.2; pages unavailable) show substantial variation across models and tasks, so the unqualified conclusion about identifying non-existing entities overstates the finding. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

HALLUHARD is a multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding.

“We introduce HALLUHARD, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding.”

• Also stated in: § 1. Introduction · p. 1 · plain

  “we introduce HALLUHARD, a new benchmark designed to evaluate hallucinations in multi-turn interactions and in more challenging task settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hallucination rates are largely stable across temperature settings.

“Overall, we find that hallucination rates are largely stable across temperature settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the judge, Serper calls for judging 10 responses cost approximately $0.11, with remaining cost from GPT-5-mini-thinking as judge model.

“For our judge, the Serper calls for judging 10 responses cost approximately $0.11, with the remaining cost coming from GPT-5-mini-thinking, which serves as the judge model.”

May combine two claims.

The judge requires roughly comparable time to agentic web search with OpenAI while costing about one-third as much.

“In addition, it requires roughly comparable time to agentic web search with OpenAI while costing about onethird as much.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the first three domains, 250 seed questions each are generated, and 200 for coding.

“For the first three domains, we generate 250 seed questions each, and 200 for coding.”

May combine two claims.

Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search, with content-grounding errors persisting at high rates.

“Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search ( ≈ 30% for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates.”

Probably not a substantial claim: may not pass S4 (based).

HALLUHARD shows high hallucination rates.

“HALLUHARD shows high hallucination rates.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Research questions are based on half niche papers (5-30 citations) and half known papers (>1000 citations).

“We base our research questions on half niche papers (between 5 and 30 citations) and half known papers (with more than 1000 citations).”

Classified as probably not a substantial claim.

Questions span four challenging domains where niche information is both prevalent and high-stakes.

“We curate questions spanning four challenging domains where niche information ('niche' refers to information that is rarely present in the data) is both prevalent and highstakes.”

• AssumptionRests on an assumption: The four domains (legal cases, research questions, medical guidelines, coding) are high-stakes and niche information is prevalent in them.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

HALLUHARD is distinguished by its multi-turn design, the inclusion of a verifiable LLM judge, and broad coverage of high-stakes domains.

“HALLUHARD is distinguished by its multi-turn design, the inclusion of a verifiable LLM judge, and broad coverage of high-stakes domains.”

May combine two claims.

Model capacity, turn position, reasoning effort, and data type all significantly affect hallucination rates.

“We find that model capacity, turn position, reasoning effort, and data type all significantly affect hallucination rates.”

May combine two claims.

Claude cites significantly more encyclopedic sources than GPT, whereas GPT more often retrieves research papers.

“We also observe a difference in web retrieval across models: Claude cites significantly more encyclopedic sources than GPT, whereas GPT more often retrieves research papers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On content grounding, most failures reflect a difference in how strictly they apply the criteria: our judge sometimes over-enforces exact terminology, while in other cases it is insufficiently strict.

“On content grounding, most failures reflect a difference in how strictly they apply the criteria: our judge sometimes over-enforces exact terminology, while in other cases it is insufficiently strict.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extractor extracted 128 claims in total.

“Total claims extracted by the extractor | 128”

Probably not a substantial claim: may not pass S3 (load-bearing).

Concerning content failures, we observe the same strictness calibration issues as with research questions.

“Concerning content failures, we observe the same strictness calibration issues as with research questions (see Example #2).”

May combine two claims.

On reference grounding, our judge is often more conservative in the presence of ambiguous bibliographic metadata, and it can reject references that humans and OpenAI-WS accept as sufficiently identified.

“On reference grounding, our judge is often more conservative in the presence of ambiguous bibliographic metadata, and it can reject references that humans and OpenAI-WS accept as sufficiently identified.”

May combine two claims.

Our judge achieves the next-highest agreement, outperforming the OpenAI web-search judge and SAFE.

“our judge achieves the next-highest agreement, outperforming the OpenAI web-search judge and SAFE”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both humans extracted 120 claims in total (intersection).

“Total claims extracted by both humans | 120”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extractor extracted 3 valid claims missed by both humans.

“Claims missed by both humans but ex- tracted by the extractor | 3”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extractor extracted 4 valid claims missed by one human.

“Claims missed by one human but extracted by the extractor | 4”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The error patterns in the legal domain are similar to those analyzed in the research questions domain.

“Overall, the error patterns in the legal domain are similar to those analyzed in the research questions domain.”

May combine two claims.

Our judge achieves the best results on content agreement and follows closely OpenAI-WS in terms of reference agreement.

“our judge achieves the best results on content agreement and follows closely OpenAI-WS in terms of reference agreement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extractor missed 0 claims that humans extracted.

“Claims extracted by humans but missed by the extractor | 0”

Probably not a substantial claim: may not pass S3 (load-bearing).

The extractor extracted 1 claim that should not have been extracted (false positive).

“Claims extracted by the extractor but should not have been extracted | 1”

May combine two claims.

LLMs struggle with niche facts, not fabricated ones; they tend to struggle with niche queries yet are more likely to abstain when faced with completely fabricated items.

“LLMs struggle with niche facts, not fabricated ones.”

• Also stated in: § 6. When Models Abstain vs Hallucinate? · p. 8 · hedged

  “models tend to struggle with niche queries, yet are more likely to abstain when faced with completely fabricated items.”

Classified as probably not a substantial claim.

SAFE works well for easy facts presented in an encyclopedia or other scholarly sources.

“This approach works well for easy facts that are presented in an encyclopedia or other scholarly sources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two post-graduate students independently extracted and annotated atomic claims from 10 responses in the research questions domain, resulting in more than 120 atomic claims.

“To assess the trustworthiness of our judging pipeline, we recruited two post-graduate students to independently extract and annotate atomic claims from 10 responses in the research questions domain, resulting in more than 120 atomic claims.”

Classified as probably not a substantial claim.

From each response, 5 claims are sampled, resulting in at most 1500 claims for judgment.

“From each response, we sample 5 claims, resulting in at most 100 × 3 × 5 = 1500 claims for judgment (For some turns, there could be less than 5 claims in total).”

May combine two claims.

The claim-sampling strategy mitigates the inflation issue by randomly sampling a fixed number of claims from each response, keeping evaluations comparable across models and obtaining a more reliable accuracy estimate.

“Our claim-sampling strategy mitigates this: by randomly sampling a fixed number of claims from each response, we keep evaluations comparable across models and obtain a more reliable accuracy estimate.”

• AssumptionRests on an assumption: Sampling 5 claims per response yields a reliable and statistically meaningful estimate of the response's hallucination rate.

May combine two claims.

As a result, SAFE may incorrectly mark a true claim as unsupported or accept a false claim if a snippet contains ambiguous phrasing.

“As a result, SAFE may incorrectly mark a true claim as unsupported simply because the relevant supporting passage is not surfaced, or conversely accept a false claim if a snippet contains ambiguous phrasing that appears to corroborate it.”

Classified as probably not a substantial claim.

A user LLM reviews previous conversation history and proposes a natural, engaging follow-up question grounded in the conversation history.

“To simulate multi-turn dialogue, we introduce a user LLM that reviews the previous conversation history and proposes a natural, engaging follow-up question grounded in the conversation history.”

• AssumptionRests on an assumption: The user LLM generates natural, engaging follow-up questions grounded in the conversation history.

Probably not a substantial claim: may not pass S3 (load-bearing).

For coding tasks, where citations are typically unavailable, the response is evaluated as a whole rather than claim by claim.

“For coding tasks, where citations are typically unavailable, we evaluate the response as a whole rather than claim by claim.”

• AssumptionRests on an assumption: In coding tasks, citations are typically unavailable.

Classified as probably not a substantial claim. May combine two claims.

To make a 250-question set, 50 comparable questions from past bar exams from The State Bar of California are generated by an LLM and reviewed and validated by a law student.

“To make a 250question set, we additionally prompt an LLM to generate 50 comparable questions from past bar exams from The State Bar of California (2025). These generated questions are subsequently reviewed and validated by a law student.”

• cites The State Bar of California (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

For the coding domain, all 200 conversations are evaluated, making 600 responses in total.

“For the coding domain, as the hallucination is measured at the response level, we evaluate all 200 conversations, making it 200 × 3 = 600 responses in total.”

May combine two claims.

Given SimpleQA's estimated benchmark error rate of about 3%, performance with web search is already near the ceiling, suggesting the benchmark is largely saturated under a browsing setting.

“Given that SimpleQA's estimated benchmark error rate is about 3% (Wei et al., 2024a), performance with web search is already near the ceiling, suggesting the benchmark is largely saturated under a browsing setting.”

• cites Wei et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The proposed judge pipeline is a modular, stage-wise system that converts assistant outputs into checkable claims, retrieves supporting evidence using Serper API, and produces structured hallucination judgments.

“Our proposed judge pipeline is a modular, stage-wise system that converts assistant outputs into checkable claims, retrieves supporting evidence using Serper API, and produces structured hallucination judgments.”

Probably not a substantial claim: may not pass S4 (based).

For legal cases, research questions, and medical guidelines, responses can be decomposed into atomic claims, so they are verified individually.

“For legal cases, research questions, and medical guidelines, responses can be decomposed into atomic claims, so we verify them individually.”

May combine two claims.

HALLUHARD mirrors real-world, open-ended interactions with LLMs while still supporting verifiable hallucination evaluation.

“HALLUHARD mirrors real-world, open-ended interactions with LLMs while still supporting verifiable hallucination evaluation.”

Classified as probably not a substantial claim. May combine two claims.

Medical guidelines are limited to existing guidelines such as NICE, publicly available prior to 2023, and questions are crafted by GPT-5-mini.

“To make sure the claims are citationverifiable, we limit ourselves to existing medical guidelines, such as NICE 1 . We download available guidelines from online PDF links and parse them into text files. We then select guidelines that are publicly available prior to 2023, and use GPT-5-mini to craft open-ended written-exam-like questions from selected guideline articles.”

• AssumptionRests on an assumption: Medical guidelines publicly available prior to 2023 are within the knowledge cutoff date of the evaluated models.

Probably not a substantial claim: may not pass S4 (based).

Most works continue to use SAFE from LongFact.

“Most works continue to use SearchAugmented Factuality Evaluator (SAFE) from LongFact.”

Classified as probably not a substantial claim.

To control evaluation cost, a subset of 100 seed questions is sampled in legal cases, research questions, and medical guidelines domains.

“To control evaluation cost, we sample a subset of 100 seed questions in the legal cases, research questions, and medical guidelines domains.”

Classified as probably not a substantial claim. May combine two claims.

In coding tasks, citing support sources is less likely, so three types of hallucinations are checked: installation, package importing, and function calling.

“Unlike the other 3 domains, citing support sources is less likely. Instead, we check the following three different types of hallucinations:”

• AssumptionRests on an assumption: In coding tasks, citations are typically unavailable.

May combine two claims.

Many existing hallucination benchmarks saturate quickly as models improve, in part because they target relatively easy domains or constrained formats such as short-form QA or classification, and because they rely on simplified single-turn prompts that diverge from real-world use.

“Many existing benchmarks saturate quickly as models improve. Yet many existing benchmarks saturate as models improve, in part because they target relatively easy domains or constrained formats such as short-form QA or classification, and because they rely on simplified single-turn prompts that diverge from realworld use.”

Classified as probably not a substantial claim.

GPT-5-mini with web search is used as the judge, instructed to review the complete output for hallucinations related to installation steps, imports, and function calls.

“We use GPT-5-mini with web search as the judge, and instruct it to review the complete output for hallucinations related to installation steps, imports, and function calls.”

Classified as probably not a substantial claim.

Each extracted claim includes both an existing reference and existing supported content.

“We ensure that each claim extracted includes both an existing reference and existing supported content.”

Classified as probably not a substantial claim.

If the cited source cannot be located, it is treated as a mis-attributed reference, an unambiguous instance of reference hallucination.

“Conversely, if the cited source cannot be located, we treat this as a mis-attributed reference, an unambiguous instance of reference hallucination.”

Classified as probably not a substantial claim.

If a response has any of the three hallucination types, the whole response is flagged as hallucination.

“If a response has any of the three hallucination types, we flag the whole response as hallucination.”

Classified as probably not a substantial claim.

For each model configuration and task domain, two follow-up questions are generated, yielding 3 turns per conversation.

“For each model configuration and task domain, we generate two follow-up questions, yielding 3 turns per conversation.”

Classified as probably not a substantial claim.

Research questions are created from abstracts of ArXiv articles published up to 2023.

“We create research questions from abstracts of ArXiv articles. To make sure that the articles are seen by the models (within their knowledge cutoff date), we only select articles that are published up to 2023.”

• AssumptionRests on an assumption: Articles published up to 2023 are within the knowledge cutoff date of the evaluated models (i.e., seen by the models).

Classified as probably not a substantial claim.

All generated seed questions are reviewed by domain experts.

“All generated seed questions are reviewed by domain experts.”

Probably not a substantial claim: may not pass S4 (based).

When a cited source is retrievable and the quoted passage can be verified, it is treated as evidence that the model is drawing from that document, enabling reliable groundedness evaluation.

“When a cited source is retrievable and the quoted passage can be verified, we treat it as evidence that the model is drawing from that document, enabling reliable groundedness evaluation.”

Classified as probably not a substantial claim.

Groundedness is operationalized by requiring inline citations for factual assertions.

“We operationalize groundedness by requiring inline citations for factual assertions.”

• Also stated in: § 1. Introduction · p. 1 · plain

  “During generation, models are instructed to support factual claims with explicit citations, providing a concrete anchor for verification.”

May combine two claims.

Existing multi-turn benchmarks largely avoid open-ended generation for easy hallucination verification, despite this being the dominant way users interact with LLMs today.

“However, these benchmarks largely avoid open-ended generation for easy hallucination verification, despite this being the dominant way users interact with LLMs today.”

Classified as probably not a substantial claim.

For both initial and follow-up questions, an explicit instruction requires the model to include inline citations supporting its factual claims.

“For both the initial (seed) question and every user LLM-generated follow-up question, we append an explicit instruction requiring the model to include inline citations that support its factual claims.”

Probably not a substantial claim: may not pass S4 (based).

Atomic-claim benchmarks typically report both accuracy (precision) and recall, since accuracy alone can be inflated when a model produces fewer, less informative claims.

“Atomic-claim benchmarks typically report both accuracy (precision) and recall, since accuracy alone can be inflated when a model produces fewer, less informative claims.”

Classified as probably not a substantial claim.

The web-search-based judge follows citations to retrieve and read referenced sources in full text, including parsing PDFs when needed.

“Our web-search-based judge then follows these citations to retrieve and read the referenced sources in full text, including parsing PDFs when needed.”

Classified as probably not a substantial claim.

For each subsequent assistant turn, the target LLM is provided with the entire dialogue to date plus the newly generated question as context.

“For each subsequent assistant turn, we provide the target LLM with the entire dialogue to date plus the newly generated question as context.”

Classified as probably not a substantial claim. May combine two claims.

The evaluator is given access to the entire response and reports response-wise hallucination rate.

“To account for the full context, we give our evaluator access to the entire response and report response-wise hallucination rate.”

May combine two claims.

The proposed judging pipeline iteratively retrieves evidence via web search, can fetch, filter, and parse full-text sources (including PDFs), and assesses whether cited material supports the generated content.

“we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content.”

May combine two claims.

LLMs produce plausible-sounding but ungrounded factual claims, and this problem worsens in multi-turn dialogue as context grows and early errors cascade.

“Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade.”

• AssumptionRests on an assumption: In practice, LLMs operate in multi-turn, open-ended conversations where context evolves, references accumulate, and early inaccuracies can propagate.

May combine two claims.

The proposed LLM judge pipeline is reliable and citation-checkable; it performs claim extraction, plans evidence retrieval, fetches full-text sources (including PDF retrieval/parsing for content-grounding verification), and produces structured reference- vs content-grounding verdicts with a fallback for hard cases.

“We propose a reliable, citation-checkable LLM judge pipeline that performs claim extraction, plans evidence retrieval, fetches full-text sources (including PDF retrieval/parsing for content-grounding verification), and produces structured reference- vs content-grounding verdicts with a fallback for hard cases.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

This claim volume provides a reliable and statistically meaningful estimate.

“This claim volume provides a reliable and statistically meaningful estimate.”

• AssumptionRests on an assumption: Sampling 5 claims per response yields a reliable and statistically meaningful estimate of the response's hallucination rate.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hallucination is often categorized into in-context hallucination and in-parameter hallucination.

“Hallucination is often categorized as two types. One is incontext hallucination, which evaluates whether a model's response is grounded in the provided context.”

• Also stated in: § 2. Related Work · p. 2 · plain

  “The other category is in-parameter hallucination, which asks whether an LLM's outputs are consistent with information encoded in its parameters.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

GPT-4o Search Preview reaches 90% accuracy on SimpleQA, and GPT-5-thinking with web search can reach 95.1% accuracy.

“For example, GPT-4o Search Preview reaches 90% accuracy on SimpleQA, and GPT-5-thinking with web search can reach 95.1% accuracy.”

May combine two claims.

Relative to SAFE, the judge can fetch full-text content from cited sources and uses an LLM to decide when evidence is sufficient rather than a fixed number of Serper calls.

“Relative to SAFE, our judge offers two key advantages: (1) it can fetch full-text content from cited sources; and (2) rather than issuing a fixed, pre-set number of Serper calls, it uses an LLM to decide when the current evidence is sufficient.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Larger models generally hallucinate less than smaller ones on response-based tasks, but this advantage does not necessarily extend to refusal-based tasks.

“Ravichander et al. (2025) find that larger models generally hallucinate less than smaller ones on response-based tasks, but this advantage does not necessarily extend to refusal-based tasks.”

• cites Ravichander et al. (2025) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S1 (asserted).

The disparity between content-grounding and reference failures reflects data availability: function names and paper titles are frequently repeated online, whereas detailed function behavior and fine-grained paper content typically appear only in full source texts, making them less prevalent in web-scale corpora.

“We hypothesize that this disparity reflects data availability : function names and paper titles are frequently repeated online, whereas detailed function behavior and finegrained paper content typically appear only in full source texts, making them less prevalent in web-scale corpora.”

Probably not a substantial claim: may not pass S1 (asserted), S4 (based). May combine two claims.

There is typically no consistent footprint in the model's training distribution for fabricated items, while niche entities often have some traces.

“We hypothesize that there's typically no consistent footprint in the model's training distribution for fabricated items, while niche entities often have some traces.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Disagreements between human annotators and our judge reflect the intrinsic difficulty of the judging task and mirror the same edge cases that also limit human annotators.

“Disagreements between human annotators and our judge reflect the intrinsic difficulty of the judging task and mirror the same edge cases that also limit human annotators.”

Classified as probably not a substantial claim.

We randomly sample 100 machine-judged claims and ask a human annotator to independently verify each decision, indicating whether they agree with the automated judge's outcome.

“To further validate our judge, we randomly sample 100 machine-judged claims and ask a human annotator to independently verify each decision, indicating whether they agree with the automated judge's outcome.”

May combine two claims.

Our evaluation highlights two settings in which hallucinations are most prevalent: (1) when models are asked about niche facts, and (2) when models try to produce detailed and in-depth claims while citing sources.

“Our evaluation highlights two settings in which hallucinations are most prevalent: (1) when models are asked about niche facts, and (2) when models try to produce detailed and in-depth claims while citing sources.”

Probably not a substantial claim: may not pass S2 (checkable).

The first failure mode (niche facts) remains intrinsically difficult because new, niche facts continually emerge.

“The first remains intrinsically difficult because new, niche facts continually emerge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A small portion of reference disagreements is attributable to lapses in bibliographic verification.

“A small portion of reference disagreements is attributable to lapses in bibliographic verification (e.g., failing to check a particular metadata field), where one annotator confirms a detail that the other overlooks.”

May combine two claims.

For both fabricated and niche items, the same question templates were used, ensuring that knowledge type is the only factor that varies.

“For both fabricated and niche items, we use the same question templates (Appendix C.4), ensuring that knowledge type is the only factor that varies.”

May combine two claims.

The model is in a dangerous middle zone: it is incentivized to guess, as there are non-zero chances of getting it correct.

“That puts the model in a dangerous middle zone: it is incentivized to guess, as there are non-zero chances of getting it correct (Kalai et al., 2025).”

• cites Kalai et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The Python result could be another evidence for 'models hallucinate more with niche knowledge'.

“The could be another evidence for 'models hallucinate more with niche knowledge'.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The annotator is given internet access and is instructed to consult external sources as needed.

“The annotator is given internet access and is instructed to consult external sources as needed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The remaining cases can be attributed to differences in strictness about terminology between the judges.

“The remaining cases can be attributed to differences in strictness about terminology between the judges, mirroring the same boundary observed in human annotations.”

Classified as probably not a substantial claim.

Two human annotators independently extracted all claims from 10 selected responses and determined whether each claim is hallucinated, considering both reference grounding and content grounding.

“We ask two human annotators to independently extract all claims from 10 selected responses and to determine whether each claim is hallucinated, considering both reference grounding and content grounding.”

Classified as probably not a substantial claim.

These annotations serve as the gold standard against which we evaluate our judge.

“These annotations serve as the gold standard against which we evaluate our judge.”

Probably not a substantial claim: may not pass S1 (asserted).

More effective thinking increases awareness of the model's knowledge boundary, leading it to abstain rather than speculate when evidence is sparse.

“We hypothesize that more effective thinking increases awareness of the model's knowledge boundary, leading it to abstain rather than speculate when evidence is sparse.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For content grounding, the primary source of disagreement reflects a stricter vs. looser standard for what counts as grounded.

“For content grounding, the primary source of disagreement reflects a stricter vs. looser standard for what counts as grounded: one annotator insists that the claim's specific terminology or phrasing appear explicitly in the source, whereas the other accepts grounding when the underlying idea is clearly supported, even if expressed in different language.”