CoChat
Sign Up
PreprintCited by 0Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareEthics and Social Impacts of AI

Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

Open access
Doshi, Savan·2026·arXiv
DOI 10.48550/arxiv.2610.24807arXiv 2610.24807
Open this paper in CoChat — free accountOpen PDF

Abstract

Patient-facing medical question answering with large language models (LLMs) is moving from research demonstrations to production deployments, yet evaluation remains anchored on factual correctness. Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient. We introduce a risk-sensitive evaluation framework that scores responses by the presence and severity of six categories of risk-bearing language (treatment directives, contraindications, dosage expressions, urgency-triage cues, high-alert medication mentions, and overconfident assertions), anchored on published clinical harm taxonomies. Severity is classified by a three-judge LLM ensemble. We apply this framework to six 7-9B instruction-tuned models, three general (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) and three medically adapted (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), on 200 real patient questions from the K-QA benchmark. Two findings emerge. First, ensemble risk-bearing language is statistically independent of factual hallucination rate (pooled Spearman $ρ= -0.065$, $n = 300$, 95% CI $[-0.18, +0.05]$; distance correlation $= 0.10$, independence test $p = 0.24$). Second, medical fine-tuning does not reduce aggregate risk but reshapes it: continued-pretraining adaptations (BioMistral, OpenBioLLM) consistently suppress high-alert medication mentions while amplifying overconfident assertions, whereas clinical instruction-tuning (Med42) suppresses dosing only, without amplifying overconfidence. Per-category Krippendorff $α$ is fair to good for the four linguistically anchored categories (0.53-0.71) and low for the two clinical-judgment-dependent categories (0.19, 0.31), indicating where automatic safety evaluation is and is not yet reliable. We release the rubric, judge prompts, and evaluation code.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper introduces a rubric-based, three-judge LLM evaluation framework for risk-bearing patient-facing medical language and applies it to six 7–9B models on 200 K-QA questions. It finds that risk-bearing language is statistically independent of factual hallucination and that medical fine-tuning reshapes rather than reduces aggregate risk.

Key findings

  • Ensemble risk-bearing language was statistically independent of factual hallucination rate: pooled Spearman ρ = −0.065, n = 300, 95% CI [−0.18, +0.05]; distance correlation = 0.10, independence test p = 0.24.
  • Aggregate risk differed negligibly across models: mean ensemble RSHS fell in [1.24, 1.43] on the 0–5 scale with overlapping 95% CIs, differing by less than 0.2.
  • Medical fine-tuning reshaped risk: continued-pretraining adaptations (BioMistral, OpenBioLLM) suppressed high-alert medication mentions while amplifying overconfident assertions (paired Wilcoxon p ≤ 0.008 in each), with BioMistral also suppressing urgency triage and OpenBioLLM suppressing dosage expression; clinical instruction-tuning (Med42) suppressed dosing only without amplifying overconfidence.
  • Judge reliability was fair to good for four linguistically anchored categories (Krippendorff α 0.53–0.71) and low for overconfident assertion (α = 0.31) and high-alert medication (α = 0.19).

Methods

  • The framework scores six non-exclusive categories of risk-bearing language—treatment directive, contraindication, dosage expression, urgency triage, high-alert medication, and overconfident assertion—on the Asgari 0–5 severity scale, using maximum-severity aggregation for overall risk and risk breadth as the count of present categories.
  • Evaluation used 200 real patient questions from K-QA, including 50 with physician-decomposed atomic must-have claims for the orthogonality experiment; six 7–9B instruction-tuned models were tested as three medical/general pairs (Mistral-7B/BioMistral-7B, Phi-3.5-mini/OpenBioLLM-Llama3-8B, Llama-3.1-8B/Med42-Llama3-8B) with 4-bit NF4 quantization, greedy decoding (seed 42, max new tokens = 300), and an identical neutral system prompt.
  • Responses were scored by a three-judge LLM ensemble (Gemini 2.5 Flash, GPT-4o-mini, Claude Haiku 3.5) at temperature 0; hallucination rate was computed with a cross-encoder NLI model, and analyses used Spearman correlation, distance correlation, bootstrap CIs, Wilcoxon signed-rank, Mann-Whitney U, and Krippendorff α.

Limitations

  • The paper states that two of six categories achieve α < 0.4, the 7–9B panel may not generalize to frontier models, generation uses greedy decoding at a single seed, and LLM-judge self-preference bias is mitigated via a three-judge cross-family ensemble; clinician-anchored calibration is ongoing and a validation subset with expert ratings is future work.
  • The text makes evident that the headline reshape finding rests on overconfident assertion, one of the two low-agreement categories; the authors treat per-response scores as uncertain while reading the directional pattern as robust, supported by per-judge consistency and a high-confidence RSHS restricted to four well-agreed categories.
  • The paper also notes that contraindication is retained as a research gap because patient-facing evidence is lacking, and that whether the reshape pattern generalizes beyond the three studied model pairs is an open question.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2610.24807

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 14 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    A recent peer-reviewed case report describes a patient who developed bromism requiring inpatient psychiatric admission after following ChatGPT-derived dietary advice that was contextually defensible but inappropriate when delivered without contraindication framing.

    “A recent peer-reviewed case report describes a patient who developed bromism requiring inpatient psychiatric admission after following ChatGPT-derived dietary advice that was contextually defensible but inappropriate when delivered without contraindication framing (n=1; Eichenberger et al. 2025)”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Eichenberger et al. 2025 source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
  • Premisecitation

    Current models assert correct and incorrect medical answers with nearly identical confidence.

    “current models assert correct and incorrect medical answers with nearly identical confidence (Omar et al., 2025;Rosenbacke et al., 2025)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Omar et al., 2025 source
    • cites Rosenbacke et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Study design and conductcitation

    The three-judge cross-family ensemble mitigates self-preference bias.

    “Three judges-Gemini 2.5 Flash, GPT-4o-mini, and Claude Haiku 3.5-are chosen for cross-family diversity to mitigate self-preference bias (Pombal et al., 2026).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Pombal et al., 2026 source
  • Premisecitation

    Current evaluation of medical LLMs is anchored almost entirely on factual correctness against reference answers.

    “Current evaluation of these systems is anchored almost entirely on factual correctness against reference answers (Agarwal et al., 2026;Pandit et al., 2025;Manes et al., 2024;Wu et al., 2024;Pal et al., 2023).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Agarwal et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Pandit et al., 2025 source

      The Introduction (p. 2) and Conclusion (p. 8) describe evaluating medical hallucination detection, and the Implementation Details (p. 5) say the benchmark uses ground-truth question–answer pairs and correctness checks. This supports a focus on factuality, but the paper does not establish that medical LLM evaluation overall is anchored almost entirely on correctness against reference answers. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Manes et al., 2024 source

      The Introduction says most existing benchmarks use textbook questions with multiple-choice or span-based answers. Sections 1 and 4 describe K-QA’s own evaluation against physician-curated reference statements for comprehensiveness and hallucinations. These passages support the emphasis on reference-answer accuracy, but do not establish that medical LLM evaluation is anchored almost entirely on it. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Rubric-prompted LLM-as-judge methods approximate clinician evaluation in adjacent settings.

    “Rubric-prompted LLM-asjudge methods have been shown to approximate clinician evaluation in adjacent settings (Arora et al., 2025)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Arora et al., 2025 source
  • Classified as probably not a substantial claim. 10
  • May combine two claims. 24
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Self-preference bias across judge-model families has been documented.

    “though self-preference bias across judge-model families has been documented (Pombal et al., 2026).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Pombal et al., 2026 source

  • May combine two claims.

    Study design and conductown data

    LLM-judge self-preference bias is mitigated via a three-judge cross-family ensemble whose individual and sub-ensemble results are stable.

    “LLM-judge self-preference bias (Pombal et al., 2026) is mitigated via a three-judge cross-family ensemble whose individual and sub-ensemble results are stable (Appendix D).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Pombal et al., 2026 source

  • Classified as probably not a substantial claim.

    Premisecitation

    The Asgari 0-5 ordinal severity scale provides the severity scale on which our rubric anchors are calibrated.

    “Severity is classified on the 0-5 ordinal scale of Asgari et al. (2025), mapped to AHRQ Common Format harm categories (Agency for Healthcare Research and Quality, 2017).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Asgari et al., 2025 source

      The Methods section (page unavailable) describes a clinical safety framework that classifies levels of harm, and the Results section says the authors evaluated the severity of major hallucinations. Neither passage identifies an Asgari 0–5 ordinal scale, so the specific scale attributed to this paper is not supported by our copy. The AHRQ mapping has a separate citation and is not assessed here. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Agency for Healthcare Research and Quality, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The eight prompts below are representative examples drawn from the 200 real patient questions used in this study (K-QA benchmark; Manes et al. 2024), and they span the four topic domains present in the benchmark.

    “The eight prompts below are representative examples drawn from the 200 real patient questions used in this study (K-QA benchmark; Manes et al. 2024).They span the four topic domains present in the benchmark.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Manes et al. 2024 source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • Rests on an assumption: The K-QA benchmark prompts are representative of real patient questions.

  • May combine two claims.

    Findingown data

    OpenBioLLM names the drug without any dose, suppressing the actionable dosing language entirely; this is the primary suppression channel for the Phi/OpenBioLLM pair.

    “OpenBioLLM names the drug without any dose, suppressing the actionable dosing language entirely.This is the primary suppression channel for the Phi/OpenBioLLM pair.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given
    § Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

    On 'If it was BV wouldn't the metrogel have worked?', Mistral-7B hedges appropriately and recommends evaluation; BioMistral responds with a definitive off-topic claim repeated twice, scoring 2.0 points higher on OVERCONFIDENT ASSERTION.

    “On”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The ensemble scores BioMistral's OVERCONFIDENT ASSERTION at 2.0 points above Mistral's while suppressing the urgency cue.

    “The ensemble scores BioMistral's OVERCONFIDENT ASSERTION at 2.0 points above Mistral's while suppressing the urgency cue.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • May combine two claims.

    Findingown data

    Mean aggregate risk score differs negligibly across the six models, all falling in [1.24, 1.43] with overlapping 95% CIs.

    “Mean ensemble RSHS differs negligibly across the six models, all falling in [1.24, 1.43] on the 0-5 scale with overlapping 95% CIs (Table 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Per-model estimates of Spearman correlation between RSHS and NLI hallucination rate are all near zero (-0.18 ≤ ρ ≤ +0.27), and no individual model reaches significance after Bonferroni correction.

    “Per-model estimates are all near zero (-0.18 ≤ ρ ≤ +0.27; Table3); no individual model reaches significance after Bonferroni correction, including the largest estimate (Phi-3.5-mini,ρ = +0.274,CI [-0.04, +0.55]).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under BH correction, 7 of 18 tests survive at FDR ≤ 0.05, including all headline reshape findings.

    “Under BH correction, 7 of 18 tests survive at FDR ≤ 0.05, including all headline reshape findings”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For Llama/Med42, Claude Haiku uniquely shows strong overconfidence suppression (∆ = -0.389, p < 0.001), while the other two judges show no effect.

    “For Llama/Med42, Claude Haiku uniquely shows strong overconfidence suppression (∆ = -0.389,p < 0.001), while the other two judges show no effect.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • May combine two claims.

    Findingown data

    Med42 suppresses the tapering schedule (dosage suppression) and appropriately deflects to clinical guidance without amplifying overconfident assertions.

    “Med42 suppresses the tapering schedule (dosage suppression) and appropriately deflects to clinical guidance without amplifying overconfident assertions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • May combine two claims.

    Findingown data

    The orthogonality finding (Section 4) is stable across individual judges (Gemini: ρ = -0.089; GPT-4o-mini: ρ = -0.079; Claude Haiku: ρ = +0.017) and all two-judge sub-ensembles (ρ ∈ [-0.102, -0.034]); none reach significance at p < 0.10, and no single judge drives the orthogonality conclusion.

    “The orthogonality finding (Section 4) is stable across individual judges (Gemini: ρ = -0.089;GPT-4o-mini: ρ = -0.079;Claude Haiku: ρ = +0.017)and all two-judge sub-ensembles (ρ ∈ [-0.102, -0.034]); none reach significance at p < 0.10.No single judge drives the orthogonality conclusion.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Phi/OpenBioLLM shows individually significant DOSAGE EXPRESSION suppression (p = 0.0003, Table 5) that is diluted in the composite mean.

    “Note that Phi/OpenBioLLM shows individually significant DOSAGE EXPRESSION suppression (p = 0.0003, Table 5) that is diluted in the composite mean.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • May combine two claims.

    Study design and conductown data

    The amplification of overconfident assertions appears in both BioMistral and OpenBioLLM pairs (paired Wilcoxon p ≤ 0.008), is consistent in at least two of three individual judges per pair, and a high-confidence RSHS computed on only the four well-agreed categories yields the same model-vs-model direction (Mistral → BioMistral: -0.172, p = 0.02).

    “the amplification appears in both BioMistral and OpenBioLLM pairs (paired Wilcoxon p ≤ 0.008), is consistent in at least two of three individual judges per pair, and a”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The continued-pretraining pattern is evidence that biomedical corpora encode a style shift-confident register replacing patient-self-management dosing language-that propagates into model behavior, and that clinical instruction-tuning on curated patient-facing dialogue does not produce.

    “We interpret the continued-pretraining pattern as evidence that biomedical corpora encode a style shift-confident register replacing patient-self-management dosing language-that propagates into model behavior, and that clinical instruction-tuning on curated patientfacing dialogue does not produce.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The risk a medical model produces is not less risk than its general counterpart-it is different risk, in ways that depend on the specific adaptation strategy.

    “The risk a medical model produces is not less risk than its general counterpart-it is different risk, in ways that depend on the specific adaptation strategy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The headline reshape finding-continued-pretraining adaptations amplifying OVERCONFIDENT ASSERTION-rests on one of the two low-α categories, so the per-response score is uncertain while the directional pattern is robust.

    “The headline reshape finding in Section 5-continued-pretraining adaptations amplifying OVERCONFIDENT ASSERTION-rests on one of the two low-α categories, so we treat the perresponse score as uncertain while reading the directional pattern as robust”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The dissenting judge differs across pairs (GPT-4o-mini for Mistral/BioMistral; Claude Haiku for Phi/OpenBioLLM), suggesting judge-idiosyncratic sensitivity rather than a systematic artifact.

    “The dissenting judge differs across pairs (GPT-4o-mini for Mistral/BioMistral; Claude Haiku for Phi/OpenBioLLM), suggesting judge-idiosyncratic sensitivity rather than a systematic artifact.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The deployment implication is direct: medical adaptation does not reduce patient risk; it reshapes it, and the adaptation strategy matters more than the binary fact of having 'medical' in the model card.

    “The deployment implication is direct: medical adaptation does not reduce patient risk; it reshapes it, and the adaptation strategy matters more than the binary fact of having”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Generation uses greedy decoding at a single seed.

    “generation uses greedy decoding at a single seed”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    For OVERCONFIDENT ASSERTION, when judges agree the assertion is present, they broadly agree on its severity, but they disagree on the detection threshold.

    “suggesting that when judges do agree the assertion is present, they broadly agree on its severity-but they disagree on the detection threshold.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Factuality is necessary but not sufficient for safe patient-facing medical AI.

    “Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    This may reflect a self-preference artifact: Med42's instruction-tuning style closely resembles Claude Haiku's hedged response pattern.

    “This may reflect a self-preference artifact: Med42's instruction-tuning style closely resembles Claude Haiku's hedged response pattern.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    This example illustrates the clinical instruction-tuning profile: risk reduction targeted at the highest-actionability category (DOSAGE EXPRESSION), without the overconfidence side effect seen in the continued-pretraining pairs.

    “This example illustrates the clinical instruction-tuning profile: risk reduction targeted at the highest-actionability category (DOSAGE EXPRESSION), without the overconfidence side effect seen in the continuedpretraining pairs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Two of six categories achieve α < 0.4.

    “Two of six categories achieve α < 0.4”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The per-category picture is different from the aggregate.

    “The per-category picture is different.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    OpenBioLLM answers factually about migraine physiology but removes the escalation cue-a form of urgency suppression that could delay care in status migrainosus.

    “OpenBioLLM answers factually about migraine physiology but removes the escalation cue-a form of urgency suppression that could delay care in status migrainosus.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.

  • May combine two claims.

    Interpretationown data

    Categories with surface lexical anchors reach the agreement ceiling of LLM judges; those requiring clinical judgment do not.

    “This reflects a real boundary on automatic rubric-based evaluation: categories with surface lexical anchors reach the agreement ceiling of LLM judges; those requiring clinical judgment do not.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The 7-9B panel may not generalize to frontier models.

    “our 7-9B panel may not generalize to frontier models”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationown data

    BioMistral introduces topical drift (genital warts, never mentioned by the patient) in a definitively stated falsehood, repeated verbatim-a pattern consistent with confident-register biomedical pretraining data contaminating the patient-facing response.

    “BioMistral introduces topical drift (genital warts, never mentioned by the patient) in a definitively stated falsehood, repeated verbatim-a pattern consistent with confident-register biomedical pretraining data contaminating the patientfacing response.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM judges' severity scores are valid measures of the risk if acted on.
31 of 31

  • The paper relies on this without support.

    The medical/general model pairs differ primarily in medical adaptation, so observed differences can be attributed to medical fine-tuning.

    “We evaluate six 7-9B instruction-tuned models as three medical/general pairs (Table 2), letting us examine whether the kind of medical adaptation shapes the risk profile.”
    “Mistral/BioMistral and Llama-3.1/Med42share base architecture; Phi-3.5/OpenBioLLM is a scale-matched comparison (3.8B vs. 8B) with different base architectures.”

  • The paper relies on this without support.

    The six categories of risk-bearing language capture the main ways patient-facing LLM responses can cause harm.

    “We define six non-exclusive categories drawn from prior clinical safety literature”
    “The rubric defines six categories of riskbearing language”

  • The paper relies on this without support.

    The severity scores assigned by the LLM judges correspond to actual patient harm severity.

    “Severity is classified on the 0-5 ordinal scale of Asgari et al. (2025), mapped to AHRQ Common Format harm categories”
    “Clinician-anchored calibration of the rubric is ongoing at the time of camera-ready submission”

  • The paper relies on this without support.

    The 200 K-QA questions are representative of real patient questions for evaluating patient-facing medical LLMs.

    “We evaluate on 200 real patient questions from K-QA (Manes et al., 2024), a benchmark of K Health platform inquiries”

  • The paper relies on this without support.

    The 200 questions provide sufficient statistical power to detect the reported effects.

    “We evaluate on 200 real patient questions from K-QA”
    “CIs are 95% percentile bootstrap (5000 resamples; 500 for α).”

  • The paper relies on this without support.

    Missed emergencies carry greater harm than unnecessary ER visits.

    “Severity is asymmetric: de-escalation of acute or potentially serious symptoms receives higher severity than over-escalation of benign symptoms, because missed emergencies carry greater harm than unnecessary ER visits.”

  • The paper relies on this without support.

    The six rubric categories capture the risk-bearing medical language relevant to patient harm.

    “The rubric covers six non-exclusive categories of risk-bearing medical language.A single statement may simultaneously match multiple categories; all present categories are scored independently.”

  • The paper relies on this without support.

    The LLM judges' severity scores are valid measures of the risk if acted on.

    “Each judge receives a system prompt specifying its evaluator role, the full severity scale, and the instruction to score risk if acted on (not factual correctness).”

  • The paper relies on this without support.

    The K-QA benchmark prompts are representative of real patient questions.

    “The eight prompts below are representative examples drawn from the 200 real patient questions used in this study (K-QA benchmark; Manes et al. 2024).”

  • The paper relies on this without support.

    The 3-judge ensemble mean is a reliable estimate of risk.

    “All ensemble RSHS values are the 3-judge mean; percategory severity values are ensemble means.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Six models described as 7-9B despite a 3.8B model

    Minor

    The study calls all six models 7-9B, but its own Table 2 description identifies Phi-3.5 as 3.8B.

  • High-confidence RSHS comparison conflicts with Table 7

    Minor

    The text gives the Mistral-to-BioMistral high-confidence comparison as -0.172 with p = 0.02; Table 7 gives -0.070 with p = 0.0076 for the same pair.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 7

  1. American Academy of Pediatrics. Metric units and the preferred dosing of orally adminis- tered liquid medications. Pediatrics, 135(4):784, 2015.
  2. Matthew Grissinger. Incorrect insulin administration: A problem that warrants attention. Diabetes Spectrum, 2016. PMC4714726. Institute for Safe Medication Practices. Ismp list of high-alert medications in com- munity/ambulatory care settings. https://www.ismp.org/system/files/resources/ 2021-09/HighAlertMedications Community-2021.pdf, 2021. Accessed: 2026-06-23.
  3. Zixuan Ji, Nayeon Lee, Rita Frieske, et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023.
  4. Yubin Kim et al. Medical hallucinations in foundation models. medRxiv preprint 2025.02.28.25323115, 2025.
  5. Eric Lehman, Yixin Wu, Yuhao Zhang, et al. Assessing the safety of large language models in clinical applications. Journal of the American Medical Informatics Association (JAMIA), 2023.
  6. Yuhao Zhang, Yixin Wu, Eric Lehman, et al. A framework to assess clinical safety and hallucination rates of large language models for medical text summarization. Journal of Biomedical Informatics, 2023.
  7. Yuyang Zhu, Qiao Chen, Haoran Wang, et al. Can we trust AI doctors? a study of medical hallucinations in large language models. In Findings of the Association for Computational Linguistics, 2025.

Related

Authors
Doshi, S.
Corresponding author
Doshi, Savan
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2610.24807
arXiv
2610.24807
OpenAlex
W7223032483
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
medical question answering, overconfidence, high-risk medications, hallucinations, instruction-tuned models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review