CoChat
Sign Up
PreprintCited by 0Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareMental Health via Writing

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

Open access
Flathers, Matthew, Nguyen, Phuong Anh, Noorily, Jill, Herpertz, Julian, Chen, Meiting, Multani, Jasreen, Powell, Samuel, Granof, Mason +2 more·2026·arXiv
DOI 10.48550/arxiv.2608.25071arXiv 2608.25071
Open this paper in CoChat — free accountOpen PDF

Abstract

General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($τ\ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.

AI snapshot

Written from the paper’s full text.

TL;DR

The authors screen OpenAI's 5,000-conversation HealthBench corpus for mental-health relevance using an LLM-applied rubric plus two rounds of blinded clinician review, releasing HealthBench-Psych (610 conversations, 12.2%) and HealthBench-Psych-Hard (n=119). Evaluating 20 models with a three-judge cross-vendor panel, they find a statistically tied frontier cluster, near-identical judge rankings (τ ≥ 0.92), and measurable refusal behavior in two models.

Key findings

  • The screen labeled the corpus RELEVANT 378 (7.6%), BORDERLINE 263 (5.3%), NOT_RELEVANT 4,359 (87.2%); round-1 review covered 791 conversations (641 screened-in + 150 concealed controls, relevance AC1 0.79), with a 6.0% control-inclusion rate that exceeded the 5% rule and triggered a recall round. The recall pass surfaced 81 candidates (9 CLEAR, 72 POSSIBLE); 14 were included and the control rate fell to 4.0%, terminating the loop. Final subset: 610 conversations (12.2%).
  • Top five models form a statistically tied frontier cluster: kimi-k2.6 (0.627), gpt-5.5 (0.624), claude-opus-5 (0.620), grok-4.5 (0.612), gpt-5.6-sol (0.610). Only kimi-k2.6 and gpt-5.5 separate from gpt-5.6-sol at uncorrected 95%, and neither survives Holm-Bonferroni correction. gpt-3.5-turbo floors at 0.176.
  • Judges ranked the 20 candidates near-identically (Kendall τ = 0.926–0.947) but differed in severity: against a grand mean of 0.524, gemini-2.5-flash graded +0.077 leniently while claude-haiku-4.5 and gpt-4.1 graded −0.045 and −0.032 strictly. Raw self-preference was large and inconsistent in sign, collapsing to within noise of zero after severity correction.
  • Two models returned empty refusals: claude-opus-5 on 10/610 conversations (1.6%) and claude-fable-5 on 3/610 (0.5%); no other model refused. Excluding refusals raises claude-opus-5 to 0.631 and claude-fable-5 to 0.594, both within the primary estimate's CI.
  • The newest frontier release in each of three lineages scored below its immediate predecessor: claude-fable-5 0.591 vs. claude-opus-5 0.620 (∆ = −0.029 [−0.045, −0.013]), kimi-k3 0.568 vs. kimi-k2.6 0.627 (−0.059 [−0.077, −0.041]), gpt-5.6-sol 0.610 vs. gpt-5.5 0.624 (−0.014 [−0.026, −0.002]).

Methods

  • Subset construction: a three-label screening rubric (RELEVANT/BORDERLINE/NOT_RELEVANT) applied by Claude Opus 4.8 in independent 50-conversation batches over the public HealthBench OSS release; all RELEVANT and BORDERLINE items plus 150 concealed NOT_RELEVANT controls were reviewed blinded by three licensed clinicians (1 MD, 1 LICSW, 1 LPC), with inclusion by ≥2/3 majority and agreement quantified by Gwet's AC1. A recall round using Claude Sonnet 4.6 re-examined excluded conversations when control inclusion exceeded 5%.
  • Response generation: one response per conversation from each of 20 models across eight providers, conversation messages verbatim with no system message, temperature 0 (provider defaults for reasoning-only models), 8,192-token cap (4,096 for gpt-3.5-turbo); refusals retained and graded as returned.
  • Grading and analysis: HealthBench grader template verbatim with a 2,048-token cap, judges at temperature 0, three judges (GPT-4.1, Claude Haiku 4.5, Gemini 2.5 Flash); scores are points for met criteria over total positive points, clipped to [0,1], with 95% CIs from conversation-level bootstrap (1,000 resamples). Harness validated by replicating OpenAI's GPT-4.1 HealthBench-Hard evaluation: 0.157 vs. published 0.16 (∆ = −0.003).

Limitations

  • Stated: the LLM screen ran inside an agentic harness with a proprietary system-prompt layer, so it is treated strictly as a prefilter with all inclusion decisions made by clinicians; concealed controls estimate roughly 6% residual mental-health content in the excluded pool by liberal clinician standards, and the recovery pass targeted high-precision misses, so some conversations may remain unrecovered; reviewers were English-speaking and rated 123 non-English conversations using machine translations; candidates were sampled once at deployed defaults, so the leaderboard reflects deployed configurations rather than matched inference compute; judge-reliability evidence derives from three judges in one specialty and is suggestive; the hard-subset comparison (n=119) has wide intervals.
  • Evident from the text: round-2 agreement was lower (AC1 0.49; remaining pair 0.84), driven by one reviewer's greater leniency on controls, which the majority rule absorbed; 65 conversations in the released subset lack a category majority; the paper notes that benchmark scores measure rubric adherence and should not be read as evidence that any model is safe or effective for mental-health support, crisis response, or clinical use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2608.25071

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 15 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    People increasingly bring mental-health concerns to general-purpose LLMs, often in moments of acute distress and outside any clinical pathway.

    “People increasingly bring mental-health concerns to general-purpose LLMs, often in moments of acute distress and outside any clinical pathway (Bodner et al., 2026;Rousmaniere et al., 2026).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Bodner et al., 2026 source

      The opening discussion reports increased interest in AI for mental health support, and the Additional studies section describes emotional conversations with Claude and ChatGPT. The paper does not establish that people often use these tools in moments of acute distress or outside a clinical pathway; it also says precise use patterns remain unclear. No page numbers are available in our copy. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Rousmaniere et al., 2026 source

      The opening section (page unavailable) says general-purpose chatbots are routinely used for mental-health questions, including by people in crisis situations, and describes a progression toward psychotherapeutic use. It does not establish that use in acute distress is frequent, or state that these interactions occur outside any clinical pathway. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation

    Existing mental-health evaluation literature clusters narrowly on suicide-risk detection and diagnostic classification.

    “Yet the existing mental-health evaluation literature clusters narrowly on suicide-risk detection and diagnostic classification (Flathers et al., 2026)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Flathers et al., 2026 source

      The Summary Findings (p. 1) report that mental-health LLM benchmarks most commonly cover risk and safety (40%), therapeutic skills (30%), and diagnosis (30%). This supports a concentration in risk and diagnosis, but the paper does not narrow those categories to suicide-risk detection and diagnostic classification, and it identifies therapeutic skills as equally common as diagnosis. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Risk classification and board-style diagnostic reasoning are poor proxies for the open-ended, ambiguous, multi-turn help-seeking that likely constitutes most real usage.

    “and risk classification and board-style diagnostic reasoning are poor proxies for the open-ended, ambiguous, multiturn help-seeking that likely constitutes most real usage (Raji et al., 2025;Stade et al., 2024;Luo et al., 2025).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Raji et al., 2025 source
    • cites Stade et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Luo et al., 2025 source

      The Results (Qualitative Results) describe varied help-seeking, including distress reduction, relationship navigation, companionship, and personal decisions; the Discussion (Principal Findings) says a majority of participants sought relief from distress. This supports the claim that real help-seeking extends beyond diagnostic tasks. The paper does not assess risk-classification or board-style tests as proxies, establish that interactions are typically multi-turn, or show that these patterns constitute most real-world usage beyond its sample. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data

    The three judges ranked the 20 candidates near-identically (Kendall τ = 0.926-0.947).

    “The three judges ranked the 20 candidates nearidentically (Kendall τ = 0.926-0.947)”
    Quote in the paperNot a substantial claim
  • Findingown data

    The top five models-kimi-k2.6 (0.627), gpt-5.5 (0.624), claude-opus-5 (0.620), grok-4.5 (0.612), and gpt-5.6-sol (0.610)-form a statistically tied frontier cluster.

    “The top five: kimi-k2.6 (0.627), gpt-5.5 (0.624), claude-opus-5 (0.620), grok-4.5 (0.612), and gpt-5.6-sol(0.610) form a statistically tied frontier cluster (CIs in Table 1).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 43
  • Classified as probably not a substantial claim. 23

  • May combine two claims.

    Premisecitation

    HealthBench carries no specialty metadata; its item tags describe conversational behavior and physician-uncertainty structure, never specific medical topic.

    “But it carries no specialty metadata.Its item tags describe conversational behavior (hedging, emergency referral, context-seeking) and physician-uncertainty structure, never specific medical topic.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Arora et al., 2025 source

      Sections 5 and 5.1 describe themes such as emergency referrals, context-seeking, and responding under uncertainty, while Section 4.3 describes physician categories tied to those themes. But the paper does not establish that HealthBench carries no specialty metadata or that its item tags never describe a medical topic; the Introduction also says conversation-specific rubrics can assess facts and misconceptions about a topic. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Study design and conductcitation

    The pipeline replicated OpenAI's published GPT-4.1 evaluation on the complete 1,000-conversation HealthBench-Hard subset, scoring 0.157 against the published 0.16 (∆ = -0.003), with 0 failed gradings and 1,000/1,000 conversations scored.

    “Replicating OpenAI's GPT-4.1 evaluation on the complete 1,000-conversation HealthBench-Hard subset using OpenAI's judging temperature of 0.5 and 2,048-token generation cap, our pipeline scores 0.157 against the published 0.16 (∆ = -0.003)(Arora et al., 2025), with 0 failed gradings and 1,000/1,000 conversations scored.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Arora et al., 2025 source

      Section 6.8 identifies HealthBench Hard as a 1,000-example subset and says the paper evaluated models on it. Our copy does not give a GPT-4.1 Hard score of 0.16 in the accompanying text; the 0.16 in Section 6.1 refers to GPT-3.5 Turbo on HealthBench overall. The pipeline’s 0.157 score and grading-completion counts are the citing authors’ own results, not results attributed to Arora et al. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Where developer-published full-HealthBench scores exist, the psychiatry subset tracks them closely: gpt-4.1 scores 0.512 (0.483 with the identical GPT-4.1 judge) here against a published 0.48, gemini-2.5-pro 0.527 (0.499) against 0.52, gpt-3.5-turbo floors at 0.176 (0.125) against 0.16, and kimi-k2.6's lead 0.627 (0.600) matches the 0.58 Moonshot reports for its reasoning predecessor.

    “Where developer-published full-HealthBench scores exist, our psychiatry subset tracks them closely: gpt-4.1 scores 0.512 (0.483 with the identical GPT-4.1 judge) here against a published 0.48, gemini-2.5-pro0.527 (0.499) against 0.52, and gpt-3.5-turbofloors at 0.176 (0.125) against 0.16, and kimi-k2.6'slead 0.627 (0.600) matches the 0.58 Moonshot reports for its reasoning predecessor (Arora et al”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Arora et al., 2025 source

      Table 7 in the Discussion reports full-HealthBench mean scores of 0.4778 for GPT-4.1 and 0.1554 for GPT-3.5 Turbo, supporting the cited comparisons to 0.48 and 0.16. Section 8.1 identifies GPT-4.1 as the default grader. Our copy evaluates Gemini 2.5 Pro but does not give its claimed 0.52 score in the retrieved text, and it does not substantiate the 0.58 Moonshot/Kimi comparison; the psychiatry-subset scores are the citing authors’ own results. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Moonshot AI, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Judges were drawn from three vendors so that judge-family effects were measurable, and all judges ran as non-reasoning models.

    “Judges were drawn from three vendors so that judge-family effects were measurable (Zheng et al., 2023;Panickssery et al., 2024), and all judges ran as nonreasoning models (Appendix E).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Zheng et al., 2023 source

      Section 3.3 compares results under different LLM judges and examines whether judges favor answers from their own models, which supports the rationale for measuring judge-related effects. It does not say that judges were drawn from three vendors; it names GPT-4, GPT-3.5, and Claude-v1 in that comparison. The separate nonreasoning-model clause cites Appendix E, not Zheng et al. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Panickssery et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The clinician-informed recall rubric re-examined the 4,209 excluded conversations not used as controls, surfacing 81 candidates (9 CLEAR, 72 POSSIBLE), which were reviewed with 25 fresh concealed controls.

    “The clinician-informed recall rubric re-examined the 4,209 excluded conversations not used as controls, surfacing 81 candidates (9 CLEAR, 72 POSSIBLE), which were reviewed with 25 fresh concealed controls.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Round-2 agreement was lower (AC1 0.49; remaining pair 0.84), driven by one reviewer's greater leniency on controls, which the majority rule absorbed.

    “Round-2 agreement was lower (AC1 0.49; remaining pair 0.84), driven by one reviewer's greater leniency on controls, which the majority rule absorbed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The two models refused disjoint conversation sets: claude-opus-5's refusals concentrated on psychiatric-medication questions and clinician-voiced requests about patient care, while all three claude-fable-5 refusals concerned neurodegeneration-related content.

    “The two models refused disjoint conversation sets: claude-opus-5's refusals concentrated on psychiatric-medication questions and clinician-voiced requests about patient care, while all three claude-fable-5 refusals concerned neurodegeneration-related content (Alzheimer's biomarkers, mechanisms, and prevalence).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority included 587 screened-in conversations and 9 of the 150 controls, a control-inclusion rate of 6.0%, which exceeded the 5% decision rule and triggered a recall round.

    “The majority included 587 screened-in conversations and 9 of the 150 controls, a control-inclusion rate of 6.0%, which exceeded the 5% decision rule and triggered a recall round.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under the primary policy (refusals graded as returned) the refused conversations scored near or below zero for the refusing model; excluding them raises claude-opus-5 to 0.631 and claude-fable-5 to 0.594, in both cases within the primary estimate's confidence interval and leaving the frontier cluster's composition unchanged.

    “Under the primary policy (refusals graded as returned) these conversations scored near or below zero for the refusing model; excluding them raises claude-opus-5 to 0.631 and claude-fable-5 to 0.594, in both cases within the primary estimate's confidence interval and leaving the frontier cluster's composition unchanged.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    HealthBench-Psych-Hard (n=119) is the intersection of this set with OpenAI's published HealthBench-Hard conversation index.

    “HealthBench-Psych-Hard (n=119) is the intersection of this set with OpenAI's published HealthBench-Hard conversation index.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across all ten paired comparisons, only kimi-k2.6 and gpt-5.5 separate from gpt-5.6-sol at uncorrected 95%, and neither separation survives Holm-Bonferroni correction.

    “Across all ten paired comparisons (Appendix F), only kimi-k2.6 and gpt-5.5 separate from gpt-5.6-sol at uncorrected 95%, and neither separation survives Holm-Bonferroni correction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The first round of expert review covered 791 conversations (641 screened-in + 150 concealed controls; relevance AC1 0.79 across the full set).

    “The first round of expert review covered 791 conversations (641 screened-in + 150 concealed controls; relevance AC1 0.79 across the full set).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The ranking on the 596-conversation round-1 subset is identical to the final n=610.

    “The ranking on the 596-conversation round-1 subset is identical to the final n=610.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    HealthBench-Psych is a clinician-adjudicated subset of 610 mental-health-relevant HealthBench conversations (12.2% of the corpus).

    “We introduce HealthBench-Psych, a clinician-adjudicated subset of 610 mental-health-relevant HealthBench conversations”
    • Also stated in: § 3.1 Subset construction and loop termination · plain
      “The released subset comprises 610 conversations (12.2% of the corpus).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Two models returned empty refusals (API stop-reason refusal, reproducible on re-query): claude-opus-5 on 10/610 conversations (1.6%) and claude-fable-5 on 3/610 (0.5%); no other model refused.

    “Two models returned empty refusals (API stopreason refusal, reproducible on re-query): claude-opus-5 on 10/610 conversations (1.6%) and claude-fable-5 on 3/610 (0.5%); no other model refused.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Below the frontier cluster, scores step down through a mid-tier (0.51-0.59) to gemini-2.5-flash, qwen3-8b, and claude-haiku-4.5 (0.44-0.46), mistral-small (0.36), and gpt-3.5-turbo (0.176).

    “Below the frontier cluster, scores step down through a mid-tier (0.51-0.59) to gemini-2.5-flash,qwen3-8b, and claude-haiku-4.5 (0.44-0.46), mistral-small (0.36), and gpt-3.5turbo(0.176).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The judges differed in severity: against a grand mean of 0.524, gemini-2.5-flash graded +0.077 leniently while claude-haiku-4.5 and gpt-4.1 graded -0.045 and -0.032 strictly.

    “but differed in severity: against a grand mean of 0.524, gemini-2.5-flashgraded +0.077 leniently while claudehaiku-4.5 and gpt-4.1 graded -0.045 and -0.032 strictly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority included 14 candidates (6/9 CLEAR, 8/72 POSSIBLE) and 1 of the 25 controls; a control-inclusion rate of 4.0%, below the stopping threshold, terminating the loop.

    “The majority included 14 candidates (6/9 CLEAR, 8/72 POSSIBLE) and 1 of the 25 controls; a control-inclusion rate of 4.0%, below the stopping threshold, terminating the loop.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Raw self-preference was large and inconsistent in sign; after severity correction it collapsed to within noise of zero for all three judges.

    “Raw self-preference was large and inconsistent in sign; after severity correction it collapsed to within noise of zero for all three judges (full analysis in Appendix I).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The newest frontier release in each leading lineage scores below its immediate predecessor: claude-fable-5 0.591 vs. claude-opus-5 0.620 (paired ∆ = -0.029 [-0.045, -0.013]), kimi-k3 0.568 vs. kimi-k2.6 0.627 (-0.059 [-0.077, -0.041]), and gpt-5.6-sol 0.610 vs. gpt-5.5 0.624 (-0.014 [-0.026, -0.002]).

    “It is notable that the newest frontier release in each leading lineage scores below its immediate predecessor.claude-fable-5 0.591 vs. claude-opus-5 0.620 (paired ∆ = -0.029[-0.045, -0.013]), kimi-k3 0.568 vs. kimi-k2.60.627 (-0.059 [-0.077, -0.041]), and gpt-5.6-sol0.610 vs. gpt-5.50.624 (-0.014 [-0.026, -0.002]).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    HealthBench-Psych contains 610 conversations.

    “Table I1: Per-judge mean scores on HealthBench-Psych (n=610).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Perinatal mental health is the largest category across both subsets, with 112 conversations (18.4%) in HealthBench-Psych and 27 conversations (22.7%) in HealthBench-Psych-Hard.

    “Perinatal mental health is the largest category across both subsets (112 conversations [18.4%]; 27 conversations [22.7%]).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The tier structure is preserved under increased difficulty, and the frontier cluster remains statistically indistinguishable in both panels.

    “The tier structure is preserved under increased difficulty, and the frontier cluster remains statistically indistinguishable in both panels.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The subset is released as conversation identifiers over the parent benchmark, adding no content beyond what HealthBench already makes public.

    “We release the subset as conversation identifiers over the parent benchmark, adding no content beyond what HealthBench already makes public.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Conversations refused by claude-opus-5 (10) and claude-fable-5 (3) on HealthBench-Psych v1.

    “Conversations refused by claude-opus-5 (10) and claude-fable-5 (3) on HealthBench-Psych v1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    HealthBench-Psych-Hard contains 119 conversations.

    “On HealthBench-Psych-Hard (n = 119) all ten intervals cross zero.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Raw self-preference was large and inconsistent in sign (haiku -0.059, gemini +0.122, gpt-4.1 -0.044).

    “Raw self-preference was large and inconsistent in sign (haiku -0.059, gemini +0.122, gpt-4.1 -0.044)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The first two gaps are decisive, the third marginal, but the direction is uniform across three lineages that post gains on general benchmarks.

    “The first two gaps are decisive, the third marginal, but the direction is uniform across three lineages that post gains on general benchmarks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    HealthBench has begun to appear in AI company model cards and technical reports as the core of their health testing, making it one of developers' most widely used mental-health evaluation instruments.

    “This matters because HealthBench has begun to appear in AI company model cards and technical reports as the core of their health testing, making it one of developers' most widely used mental-health evaluation instruments (Ope-nAI, 2026; Anthropic, 2026; Meta Superintelligence Labs, 2026).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites OpenAI, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Anthropic, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Meta Superintelligence Labs, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Every conversation-candidate pair was graded independently by a panel of three judges: GPT-4.1, Claude Haiku 4.5, and Gemini 2.5 Flash.

    “Every conversation-candidate pair was graded independently by a panel of three judges: GPT-4.1 (the grader used by HealthBench itself), Claude Haiku 4.5, and Gemini 2.5 Flash.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    While the three-vendor panel ranked models near-interchangeably (τ ≥ 0.92), judges differed substantially in severity.

    “While our three-vendor panel ranked models near-interchangeably (τ ≥ 0.92), judges differed substantially in severity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The rubric text was drafted with LLM assistance, with the inclusion and exclusion criteria set, reviewed, and finalized by the authors.

    “The rubric text was drafted with LLM assistance (Claude Fable 5), with the inclusion and exclusion criteria set, reviewed, and finalized by the authors.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Responses were stored keyed by conversation, and every judge graded the identical stored response to isolate judge effects from sampling variance.

    “Responses were stored keyed by conversation, and every judge graded the identical stored response to isolate judge effects from sampling variance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Judges graded at temperature 0 rather than the default 0.5, trading the reference's sampling regime for deterministic, reproducible verdicts.

    “We departed from the reference implementation in one parameter: judges graded at temperature 0 rather than the default 0.5, trading the reference's sampling regime for deterministic, reproducible verdicts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The LLM screen was executed inside an agentic orchestration harness that adds a proprietary system-prompt layer; the authors treat it strictly as a prefilter, with all inclusion decisions made by clinician review, and report it as LLM-assisted candidate generation rather than a controlled inference procedure.

    “The LLM screen was executed inside an agentic orchestration harness that adds a proprietary systemprompt layer; we therefore treat it strictly as a prefilter, with all inclusion decisions made by clinician review, and report it as LLM-assisted candidate generation rather than a controlled inference procedure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Judge-reliability evidence derives from three judges in one specialty and should be treated as suggestive.

    “Judgereliability evidence derives from three judges in one specialty and should be treated as suggestive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Inclusion was by reviewer majority (≥ 2/3), with per-item votes and a consensus tier retained so the release preserves disagreement.

    “Inclusion was by reviewer majority (≥ 2/3), with per-item votes and a consensus tier retained so the release preserves disagreement.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Empty responses returned with an explicit refusal stop-reason were retained as the model's response and graded as returned.

    “Empty responses returned with an explicit refusal stop-reason were retained as the model's response and graded as returned.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Grading reused the HealthBench grader verbatim: the exact grader template, the original system message, and the original 2,048-token grader cap, with each rubric criterion graded by an independent judge call returning a boolean criteria_met verdict.

    “Grading reused the HealthBench grader verbatim: the exact grader template (Appendix C), the original system message, and the original 2,048-token grader cap, with each rubric criterion graded by an independent judge call returning a boolean criteria_met verdict.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Reviewers were English-speaking and rated 123 non-English conversations with machine translations as reference.

    “Reviewers were Englishspeaking and rated 123 non-English conversations with machine translations as reference.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The screening rubric maps each conversation, from its user turns alone, to one of three labels: RELEVANT, BORDERLINE, or NOT_RELEVANT.

    “We authored a screening rubric (Appendix A) that maps each conversation, from its user turns alone, to one of three labels: RELEVANT, BORDERLINE, or NOT_RELEVANT.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Each candidate received the HealthBench conversation messages verbatim, with no added system message and no prompt modification.

    “Each candidate received the HealthBench conversation messages verbatim, with no added system message and no prompt modification.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Ties resolved to the less-inclusive label, making the screen a deliberately conservative, precision-oriented first stage.

    “Ties resolved to the less-inclusive label, making the screen a deliberately conservative, precision-oriented first stage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Uncertainty was estimated by conversation-level bootstrap (1,000 resamples), following the reference's bootstrap recipe.

    “Uncertainty was estimated by conversation-level bootstrap (1,000 resamples), following the reference's bootstrap recipe.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    All run artifacts (model snapshots, decoding parameters, prompts, grader-template hash, subset content hashes, spend and error logs) are recorded in released manifests.

    “All run artifacts (model snapshots, decoding parameters, prompts, grader-template hash, subset content hashes, spend and error logs) are recorded in released manifests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The evaluation shows that frontier models are presently inseparable on mental-health conversations while differing measurably in refusal behavior.

    “and its evaluation shows that frontier models are presently inseparable on mental-health conversations while differing measurably in refusal behavior.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Three licensed clinicians reviewed a blinded set comprising all RELEVANT and BORDERLINE conversations plus 150 NOT_RELEVANT conversations interleaved as concealed controls, in a single seeded random order with screen labels withheld, each independently recording relevance, category, confidence, and free-text notes.

    “Three licensed clinicians (R1-R3; 1 MD, 1 LICSW, 1 LPC) then reviewed a blinded set comprising all RELEVANT and BORDERLINE conversations plus 150 NOT_RELEVANT conversations in-terleaved as concealed controls, in a single seeded random order with screen labels withheld, each independently recording relevance (binary), category, confidence, and free-text notes (instrument in Appendix D).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Absolute scores were comparable only within a single judge, or after severity correction.

    “Absolute scores were comparable only within a single judge, or after severity correction”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Empty responses attributable to transport failure were re-requested and never persisted as model output.

    “Empty responses attributable to transport failure were re-requested and never persisted as model output.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Because responses, grades, and analysis code are released, the leaderboard can be re-graded under new judges and extended to new models without regeneration, and the pipeline transfers to other specialties within HealthBench and beyond.

    “Because responses, grades, and analysis code are released, the leaderboard can be re-graded under new judges and extended to new models without regeneration, and the pipeline transfers to other specialties within HealthBench and beyond.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Candidates were sampled once at deployed defaults, so the leaderboard reflects deployed configurations and not matched inference compute, and reasoning-by-default models spend variable compute.

    “Candidates were sampled once at deployed defaults, so the leaderboard reflects deployed configurations and not matched inference compute, and reasoning-bydefault models spend variable compute.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    A control-inclusion rate above 5% triggered a recall round; below it, the included set is accepted as comprehensive and the cycle terminates.

    “A control-inclusion rate above 5% triggered a recall round; below it, the included set is accepted as comprehensive and the cycle terminates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The concealed controls estimate roughly 6% residual mental-health content in the excluded pool by liberal clinician standards; the recovery pass targeted high-precision misses, so some conversations may remain unrecovered.

    “The concealed controls estimate roughly 6% residual mental-health content in the excluded pool by liberal clinician standards; the recovery pass targeted high-precision misses, so some conversations may remain unrecovered.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The authors did not re-sample refusals, since re-rolling until compliance would misrepresent deployed behavior, and, at temperature 0, would be likely to recur.

    “We did not re-sample refusals, since re-rolling until compliance would misrepresent deployed behavior, and, at temperature 0, would be likely to recur.”
    Quote in the paperSubstantial claim
54 of 54

  • The paper relies on this without support.

    The HealthBench rubric criteria are a valid measure of mental-health conversational quality.

    “A response's score followed HealthBench's scoring convention: points for met criteria over total positive points, computed with the reference scorer.”
    “Benchmark scores measure rubric adherence on fixed conversations and should not be read as evidence that any model is safe or effective for mental-health support, crisis response, or clinical use”

  • The paper relies on this without support.

    The clinician majority vote is a valid gold standard for mental-health relevance.

    “Inclusion was by reviewer majority (≥ 2/3), with per-item votes and a consensus tier retained so the release preserves disagreement.”

  • The paper relies on this without support.

    The three LLM judges are adequate proxies for expert evaluation of mental-health conversations.

    “Every conversation-candidate pair was graded independently by a panel of three judges: GPT-4.1 (the grader used by HealthBench itself), Claude Haiku 4.5, and Gemini 2.5 Flash.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported per-conversation panel scores fall below the stated clipping range

    Minor

    The scoring description says per-conversation panel scores are clipped to [0, 1], but the refusal appendix labels negative values as per-conversation panel scores.

1 of 1

Cites

Flagged first, then by citations

3 of the 17 references couldn’t be matched to a known paper.

Related

Authors
Flathers, M., Nguyen, P. A., Noorily, J., Herpertz, J., Chen, M., Multani, J., Powell, S., Granof, M., Kalinch, M., & Torous, J.
Affiliations
Beth Israel Deaconess Medical Center (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2608.25071
arXiv
2608.25071
OpenAlex
W7204449673
Cited by
0
References
17
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
Keywords
mental health, large language models, benchmark dataset, LLM evaluation, LLM-as-a-judge, psychological support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review