CoChat
Sign Up
PreprintCited by 14Artificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)Machine Learning in Healthcare

Trustworthy Medical Question Answering: An Evaluation-Centric Survey

Open access
Wang, Yinuo, Wang, Baiyang, Mercer, Robert E., Rudzicz, Frank, Roy, Sudipta Singha, Ren, Pengjie, Chen, Zhumin, Wang, Xindi·2025·arXiv
DOI 10.48550/arxiv.2506.03659arXiv 2506.03659
Open this paper in CoChat — free accountOpen PDF

Abstract

Trustworthiness in healthcare question-answering (QA) systems is important for ensuring patient safety, clinical effectiveness, and user confidence. As large language models (LLMs) become increasingly integrated into medical settings, the reliability of their responses directly influences clinical decision-making and patient outcomes. However, achieving comprehensive trustworthiness in medical QA poses significant challenges due to the inherent complexity of healthcare data, the critical nature of clinical scenarios, and the multifaceted dimensions of trustworthy AI. In this survey, we systematically examine six key dimensions of trustworthiness in medical QA, i.e., Factuality, Robustness, Fairness, Safety, Explainability, and Calibration. We review how each dimension is evaluated in existing LLM-based medical QA systems. We compile and compare major benchmarks designed to assess these dimensions and analyze evaluation-guided techniques that drive model improvements, such as retrieval-augmented grounding, adversarial fine-tuning, and safety alignment. Finally, we identify open challenges-such as scalable expert evaluation, integrated multi-dimensional metrics, and real-world deployment studies-and propose future research directions to advance the safe, reliable, and transparent deployment of LLM-powered medical QA.

AI snapshot

Written from the paper’s full text.

TL;DR

This evaluation-centric survey reviews trustworthiness in LLM-based medical question answering across six dimensions: factuality, robustness, fairness, safety, explainability, and calibration. It compiles benchmarks and evaluation-guided improvement techniques, then identifies open challenges for safe, reliable, and transparent deployment.

Key findings

  • The survey defines six core trustworthiness dimensions for medical QA and maps evaluation methods and representative benchmarks for each, including Med-HALT, MedSafetyBench, MedExQA, BiasMedQA, MedFuzz, TrustLLM, and HELM.
  • It highlights cross-dimension interactions, such as factuality–calibration, robustness–factuality, fairness–safety, explainability–calibration, and calibration–safety; for example, it cites atypicality-aware recalibration reducing ECE by 60% as a calibration improvement that can yield safer behavior.
  • It reports that Med-PaLM achieved a passing score of 67.6% on USMLE-style MedQA questions and Med-PaLM 2 reached 86.5% accuracy, while noting persistent trust failures such as hallucinated medical facts, unjustified overconfidence, and biased or unsafe recommendations.
  • Evaluation-guided improvements discussed include retrieval-augmented grounding, adversarial fine-tuning, fairness-aware data and prompt design, safety alignment via SFT/RLHF/DPO, chain-of-thought and self-reflection for explainability, and post-hoc calibration.

Methods

  • Design: an evaluation-centric survey of trustworthiness in medical QA, organizing six dimensions—Factuality, Robustness, Fairness, Safety, Explainability, and Calibration—into a unified taxonomy.
  • Data/sample: literature review; the paper states it excluded publications on general-domain LLMs and healthcare-related literature not directly applicable to medical QA tasks.
  • Analysis: compares major benchmarks and tools, analyzes evaluation-guided techniques that drive model improvements, and synthesizes open challenges and future directions; no primary experiments or new empirical data are reported.

Limitations

  • Paper-stated: the study specifically focuses on medical QA systems and excluded general-domain LLM publications as well as healthcare-related literature not directly applicable to medical QA tasks.
  • Evident from the text: as a survey, it presents no primary experiments, so its conclusions depend on the reviewed literature and the coverage of existing benchmarks.
  • The paper also states that current evaluations remain limited, many essential qualities are difficult to quantify, and existing benchmarks inadequately capture real-world complexity; challenges include scalable expert evaluation, narrow scenario/specialty/language coverage, isolated dimension evaluation, and the gap between static benchmarks and real-world deployment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.03659

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

38 of 73 cited sources checked · 20 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Med-PaLM 2 reached 86.5% accuracy on USMLE-style MedQA questions.

    “Med-PaLM 2 reached 86.5% accuracy”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Singhal et al., 2023 source

      The ‘State of the art on MedQA’ section reports 67.6% accuracy for Flan-PaLM 540B on USMLE-style questions, not 86.5% for Med-PaLM 2. Searches for ‘Med-PaLM 2’ and ‘86.5%’ found no passage in our copy reporting the claimed result. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Premisecitation

    The Med-PaLM-2 study specifically included a set of adversarial questions designed to probe the limitations of LLMs.

    “the Med-PaLM-2 study specifically included a set of adversarial questions designed to probe the limitations of LLMs”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Singhal et al., 2025 source
  • Premisecitation

    Complementary studies expose model tendencies to perpetuate debunked race-based practices.

    “Complementary studies expose model tendencies to perpetuate debunked racebased practices (Omiye et al., 2023)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Omiye et al., 2023 source
  • Premisecitation

    Adversarial benchmarks like Med-HALT and targeted false-confidence probes stress-test models with challenging prompts designed to induce fabrications, thereby quantifying a model's propensity to hallucinate under duress.

    “Adversarial benchmarks like Med-HALT (Pal et al., 2023) and targeted”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Pal et al., 2023 source
  • Premisecitation

    Models fine-tuned to resist adversarial or paraphrased inputs demonstrate lower hallucination rates, since they rely less on spurious patterns.

    “Models fine-tuned to resist adversarial or paraphrased inputs (e.g., via MedFuzz-style perturbations) demonstrate lower hallucination rates, since they rely less on spurious patterns (Asgari et al., 2025).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Asgari et al., 2025 source

      The Results section reports lower hallucination rates after changes to structured prompts and writing-style guidance for clinical-note summarisation. It does not report fine-tuning against adversarial or paraphrased inputs, MedFuzz-style perturbations, or reduced reliance on spurious patterns; those explanations are unsupported by this paper. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 60
  • May combine two claims. 38
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Med-PaLM achieved a passing score of 67.6% on USMLE-style MedQA questions.

    “Med-PaLM achieved a passing score of 67.6% on USMLE-style MedQA questions”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Singhal et al., 2023 source

      The “State of the art on MedQA” section attributes 67.6% accuracy on USMLE-style MedQA questions to Flan-PaLM 540B, not Med-PaLM. The Main section distinguishes Med-PaLM as a subsequently adapted model. The cited result is described as accuracy, not a passing score; no page number is available in our copy. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Alonso et al. (2024) included human annotation in MedExQA and demonstrated that models offering better explanation correlated with deeper understanding.

    “Alonso et al. (2024) included human annotation in MedExQA and demonstrated that models offering better explanation correlated with deeper understanding.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Alonso et al., 2024 source

      The Introduction (p. 3) says MedExpQA includes reference explanations written by medical doctors and uses a dataset annotated in multiple languages, supporting the human-annotation part. The Concluding Remarks (p. 19) say evaluating the quality of explanations generated by LLMs is future work; the paper does not demonstrate a correlation between better model explanations and deeper understanding. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Alberts et al. (2023) emphasized that adversarial testing in medical QA must account for the inherent complexity of the domain, noting that even slight modifications in phrasing can significantly alter clinical interpretations.

    “Alberts et al. (2023) emphasized that adversarial testing in medical QA must account for the inherent complexity of the domain, noting that even slight modifications in phrasing can significantly alter clinical interpretations.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Alberts et al., 2023 source

      In the paper’s discussion of its nuclear-medicine question test (page unavailable), Alberts et al. describe ChatGPT as context-sensitive and report different, incorrect answers after a question was rephrased. They also discuss adversarial examples and the difficulty of constructing them for LLMs. They do not say that adversarial testing must account for the inherent complexity of medical QA, or establish that slight phrasing changes significantly alter clinical interpretations; their example involved both rephrasing and corrective feedback. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Xian et al. (2024) develops a query-efficient adversarial sampling method, which leverages power-scaled distance-weighted sampling (PDWS) to generate realistic adversarial distractions in clinical queries, effectively testing robustness under adversarial conditions.

    “Xian et al. ( 2024) develops a query-efficient adversarial sampling method, which leverages power-scaled distanceweighted sampling (PDWS) to generate realistic adversarial distractions (e.g., disease and pharmaceutical entities) in clinical queries, effectively testing robustness under adversarial conditions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Xian et al., 2024 source

  • May combine two claims.

    Premisecitation

    EquityMedQA introduces seven adversarial datasets and human evaluation rubrics to measure disparities across race, gender, and geography, revealing subtle inequities in LLM responses.

    “EquityMedQA introduces seven adversarial datasets and human evaluation rubrics to measure disparities across race, gender, and geography, revealing subtle inequities in LLM responses (Pfohl et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Pfohl et al., 2024 source

  • May combine two claims.

    Premisecitation

    Almanac uses RAG frameworks to convert clinical QA tasks into search and retrieval processes, which use LLMs for knowledge distillation from authoritative medical sources to minimize hallucination risks.

    “Almanac (Zakka et al., 2024) uses RAG frameworks to convert clinical QA tasks into search and retrieval processes, which use LLMs for knowledge distillation from authoritative medical sources to minimize hallucination risks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zakka et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MedExQA involves human scoring of explanation correctness.

    “Human evaluation remains crucial in many benchmarks -MultiMedQA's 12-axis rubric is administered by clinicians to rate each answer qualitatively (Singhal et al., 2025), and MedExQA involves human scoring of explanation correctness (Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

  • May combine two claims.

    Premisecitation

    Automated evaluations using content filters or classifiers can detect overtly harmful responses, but nuanced medical contexts require human expert reviews.

    “Automated evaluations using content filters or classifiers can detect overtly harmful responses, but nuanced medical contexts require human expert reviews.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chowdhury et al., 2023 source

      Section 3.3 (page not provided) says two ophthalmologists assessed the responses for helpfulness, clinical harm, and appropriateness, supporting the role of expert review in this medical context. The paper does not report that automated content filters or classifiers can detect overtly harmful responses, or compare their performance with expert review. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    When deployed in clinical settings, LLM-based medical QA models continue to exhibit critical trust failures including hallucinated medical facts, unjustified overconfidence, and occasional biased or unsafe recommendations.

    “when deployed in clinical settings, these models continue to exhibit critical trust failures: hallucinated medical facts, unjustified overconfidence, and occasional biased or unsafe recommendations (Aljohani et al., 2025).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Aljohani et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The limitations of ROUGE/BLEU are acknowledged.

    “ROUGE/BLEU are used for comparing generated text with reference comparison, but their limitations are acknowledged (Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

      Section 4.4 (page unavailable) says BLEU and ROUGE compare generated explanations with reference explanations, supporting their use for reference-based comparison. Section 5.4 reports that human evaluations contrasted with other evaluation methods, but the paper does not explicitly identify limitations of ROUGE or BLEU; its Limitation section addresses the benchmark and the small scale of human evaluation instead. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Expert human review remains the gold standard for factuality evaluation, with clinicians applying structured rubrics to rate answers on accuracy, completeness, and consistency with medical consensus.

    “expert human review remains the gold standard: clinicians apply structured rubrics (for example, the Med-PaLM evaluation framework (Sing-hal et al., 2023)) to rate answers on accuracy, completeness, and consistency with medical consensus.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Singhal et al., 2023 source

  • May combine two claims.

    Premisecitation

    Benchmarks such as MedExQA explicitly require models to provide explanations, comparing them against multiple ground-truth explanations using lexical metrics.

    “Benchmarks such as MedExQA (Kim et al., 2024) explicitly require models to provide explanations, comparing them against multiple ground-truth explanations using lexical metrics (e.g., BLEU/ROUGE).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

  • May combine two claims.

    Premisecitation

    Benchmarks frequently allow lenient scoring or use multiple expert-generated references, as in MedExQA's ensemble of clinician explanations.

    “benchmarks frequently allow lenient scoring or use multiple expert-generated references, as in MedExQA's ensemble of clinician explanations (Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

      Section 3.1 (page unavailable) says MedExQA provides two explanations per question, drawn in part from online tests and exams and then human-validated. It does not identify them as clinician-generated. The Introduction (page unavailable) says existing benchmarks often lack explanations, so this paper does not support the broader claim that benchmarks frequently use multiple expert-generated references or lenient scoring. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CardioCanon, a cardiology-focused chatbot, leverages RAG to ensure the accuracy and reliability of cardiological responses.

    “CardioCanon, a cardiology-focused chatbot, leverages RAG to ensure the accuracy and reliability of cardiological responses (Tran et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Tran et al., 2024 source

      The CardioCanon abstract says the cardiology-focused chatbot uses RAG to retrieve relevant documents and reports better scores than ChatGPT 3.5 on an assessment of response quality. It describes RAG as improving performance, not ensuring the accuracy or reliability of every cardiological response. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Because factuality in medicine can sometimes be a grey area, especially when clinical guidelines evolve or expert consensus varies, automated metrics alone may not suffice.

    “Because factuality in medicine can sometimes be a”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Landsheer, 2018 source

      The Introduction (page unavailable) discusses inconclusive diagnostic test results, not whether medical statements are factually correct. Searches for evolving clinical guidelines, varying expert consensus, and automated factuality metrics found no passage addressing those claims. (LLM, conf=0.75)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ROUGE/BLEU are used for comparing generated text with reference comparison.

    “ROUGE/BLEU are used for comparing generated text with reference comparison, but their limitations are acknowledged (Kim et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kim et al., 2024 source

  • May combine two claims.

    Premisecitation

    MedSafetyBench supplies standardized unsafe scenarios that highlight failure modes and serve as a gold standard for measuring and guiding improvements.

    “MedSafetyBench supplies standardized unsafe scenarios that highlight failure modes and serve as a gold standard for measuring and guiding improvements (Han et al., 2024a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Han et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    A model trained primarily on formal medical texts may struggle with questions phrased in layperson language.

    “a model trained primarily on formal medical texts may struggle with questions phrased in layperson language.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    QA-level calibration frameworks extend conventional reliability diagrams to entire question-answer groupings, offering theoretical guarantees that underlie more robust confidence estimates.

    “QA-level calibration frameworks extend conventional reliability diagrams to entire question-answer groupings, offering theoretical guarantees that underlie more robust confidence estimates (Mastakouri et al., 2025).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mastakouri et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    TrustLLM Benchmark is an integrated toolkit that aggregates over 18 evaluation categories for LLMs, including medical QA scenarios.

    “For example, the TrustLLM Benchmark is an integrated toolkit that aggregates over 18 evaluation categories for LLMs, including medical QA scenarios (Huang et al., 2024b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Sample-consistency methods that prompt the model to generate and compare multiple reasoning chains both improve calibration and produce more faithful explanations.

    “sample-consistency methods-prompting the model to generate and compare multiple reasoning chains-both improve calibration and produce more faithful explanations (Savage et al., 2024b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Savage et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MedSafety-Bench incorporates a safety score metric.

    “To capture trust facets, some benchmarks incorporate custom metrics: e.g., Med-HALT's false confidence rate (Pal et al., 2023), or MedSafety-Bench's safety score (Han et al., 2024a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Han et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Yang et al. (2024b) employs adversarial methods via prompt engineering and fine-tuning, which highlights model vulnerabilities and notes significant impacts of fine-tuning adversarial attacks on model weights.

    “Yang et al. (2024b) employs adversarial methods via prompt engineering and fine-tuning, which highlights model vulnerabilities and noting significant impacts of finetuning adversarial attacks on model weights, an observation meriting further exploration.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    MedSafety-Bench shows that LLMs often fail safety standards and demonstrate improvements through fine-tuning.

    “It shows that LLMs often fail safety standards and demonstrate improvements through fine-tuning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Han et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Thirunavukarasu et al. (2023) proposed evaluating the variance in model outputs across semantically equivalent inputs as an indicator of robustness.

    “Thirunavukarasu et al. (2023) proposed evaluating the variance in model outputs across semantically equivalent inputs as an indicator of robustness.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Thirunavukarasu et al., 2023 source

  • May combine two claims. The quote only partly states this claim.

    Premisecitation

    Prompt engineering and user-centric guidance can nudge models toward equitable outputs, with follow-up studies showing prompt designs that specifically address cognitive biases.

    “prompt engineering (e.g.,”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Schmidgall et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Cognitive biases embedded in user inputs can distort model outputs, an effect quantified by BiasMedQA through bias-laden prompts and error analysis.

    “cognitive biases embedded in user inputs can distort model outputs-an effect quantified by BiasMedQA through bias-laden prompts and error analysis (Schmidgall et al., 2024a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Schmidgall et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Zhang et al. (2023) introduces a Let's think step by step approach specifically to improve medical reasoning, which evaluation shows reduced incorrect answers and makes reasoning transparent.

    “Zhang et al. (2023) introduces”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Traditional metrics such as accuracy and precision/recall are standard for factual correctness across these benchmarks.

    “Common Evaluation Metrics Across these benchmarks, traditional metrics such as accuracy and precision/recall are standard for factual correctness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    TrustLLM provides a unified pipeline to test a model on many trust dimensions and compare results.

    “It provides a unified pipeline to test a model on many trust dimensions and compare results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Studies of demographic bias in medical LLMs show that fairness interventions such as adversarial debiasing reduce both performance gaps and harmful, biased recommendations.

    “Studies of demographic bias in medical LLMs show that fairness interventions (such as adversarial debiasing) reduce both performance gaps and harmful, biased recommendations (Walsh et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Walsh et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Evaluation metrics from synthetic question studies on TREC LiveQA and MedRedQA reveal gaps between automated scores and human judgments, underscoring the need for nuanced, human-informed assessments.

    “Evaluation metrics from synthetic question studies on TREC LiveQA and MedRedQA further reveal gaps between automated scores and human judgments, underscoring the need for nuanced, human-informed assessments (Diekmann et al., 2025).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Diekmann et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Medical explainability QA benchmark covers 5 underrepresented specialties, such as speech pathology and clinical psych, with multiple ground-truth explanations per Q&A.

    “Medical explainability QA benchmark.Covers 5 underrepresented specialties (e.g.speech pathology, clinical psych) with multiple ground-truth explanations per Q&A.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    MedXpertQA is a comprehensive benchmark for assessing expert-level medical knowledge and advanced reasoning capabilities, comprising Text and MM subsets, with an independently designed reasoning subset.

    “MedXpertQA A comprehensive benchmark for assessing expert-level medical knowledge and advanced reasoning capabilities, comprising Text (text-based) and MM (multimodal) subsets, with an independently designed reasoning subset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    BiasMedQA introduces common clinically relevant cognitive biases into USMLE questions to test the performance of the model when facing these biases.

    “Fairness: Introduce common clinically relevant cognitive biases into USMLE questions to test the performance of the model when facing these biases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Expert review ensures high-quality critique.

    “Expert review ensures high-quality critique, but it cannot scale to the volume of queries real systems face, and inter-rater consistency varies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    It evaluates if models can provide nuanced medical explanations beyond just correct answers.

    “Explainability: evaluates if models can provide nuanced medical explanations beyond just correct answers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    A Medical Reasoning Evaluation Benchmark for LLMs combines expert-annotated and automated knowledge expansion, designed to assess contextual reasoning capabilities across medical texts and domain knowledge.

    “A Medical Reasoning Evaluation Benchmark for LLMs that Combine Expert-Annotated and Automated Knowledge Expansion, designed to assess contextual reasoning capabilities across medical texts and domain knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Med-HALT is a medical hallucination test dataset derived from medical exams across countries to probe factual recall and reasoning.

    “Med-HALT Medical Hallucination Test dataset. Derived from medical exams across countries to probe factual recall and reasoning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    By applying adversarial perturbations to medical question-answering queries, the benchmark evaluates the robustness and performance of large language models in medical question-answering tasks.

    “By applying adversarial perturbations to medical question-answering queries, evaluate the robustness and performance of large language models (LLMs) in medical question-answering tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    For each question in MedExpQA, a standard answer is provided along with multiple Gold-Explanation explanations written by medical experts.

    “For each question, a standard answer is provided along with multiple Gold-Explanation explanations written by medical experts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    MedHallBench designs questions centered around object hallucinations, attribute hallucinations, multimodal conflict hallucinations, and logical reasoning hallucinations, and conducts adversarial tests to uncover causes of hallucinations in models.

    “Hallucination: Design questions centered around object hallucinations, attribute hallucinations, multimodal conflict hallucinations, and logical reasoning hallucinations, and conduct adversarial tests to uncover the causes of hallucinations in models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Given a question and a medical text context with the conclusion section removed, the benchmark evaluates whether the model can infer if the question originally appeared in the conclusion section of the source text.

    “Reasoning: Given a question and a medical text context with the conclusion section removed, evaluate whether the model can infer if the question originally appeared in the conclusion section of the source text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    For explainability, some tools allow automated reasoning verification, such as checking chain-of-thought logic or using another LLM to critique the answer's reasoning.

    “For explainability, some tools allow automated reasoning verification, such as checking chain-of-thought logic or using another LLM to critique the answer's reasoning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Existing benchmarks inadequately capture real-world complexity.

    “However, current evaluations remain limited; many essential qualities are difficult to quantify, and existing benchmarks inadequately capture real-world complexity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationnone given

    Evaluation not only measures performance but also provides critical feedback to drive improvements.

    “A key insight is that evaluation is not only measures performance but also provides critical feedback to drive improvements.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Evaluating trustworthiness in medical QA systems involves multiple dimensions, including factuality, robustness, fairness, safety, explainability, and calibration.

    “Evaluating trustworthiness in medical QA systems involves multiple dimensions, including factuality, robustness, fairness, safety, explainability, and calibration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The first binary classification benchmark for medical hallucination detection.

    “The first binary classification benchmark for medical hallucination detection.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    It evaluates whether models can ensure response integrity when handling inputs containing unsafe medical instructions, as benchmarked by MedSafetyBench's adversarial testing framework.

    “Safety: Evaluate whether models can ensure response integrity when handling inputs containing unsafe medical instructions, as benchmarked by MedSafetyBench's adversarial testing framework.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The first medical-domain Safety evaluation benchmark dataset focused on assessing model responses to unsafe medical instructions.

    “The first medical-domain Safety evaluation benchmark dataset focused on assessing model responses to unsafe medical instructions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    We lack frameworks to jointly evaluate these trade-offs or to report composite trustworthiness metrics.

    “We lack frameworks to jointly evaluate these trade-offs or to report composite trustworthiness metrics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Med-HALT evaluates how often models produce unsupported info under stress-test conditions.

    “Evaluates how often models produce unsupported info under stress-test conditions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The questions are divided into three levels-Easy, Medium, and Hard-according to the difficulty of identifying hallucinations.

    “The questions are divided into three levels -Easy, Medium, and Hardaccording to the difficulty of identifying hallucinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Med-HALT includes reasoning-based tests and memory-based recall tests to quantify hallucination rates.

    “Hallucination: includes reasoning-based tests (”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Projects like MedExQA, which added speech pathology, demonstrate the value of domain expansion.

    “Projects like MedExQA, which added speech pathology, demonstrate the value of domain expansion-but many fields remain untested.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Many essential qualities are difficult to quantify.

    “However, current evaluations remain limited; many essential qualities are difficult to quantify, and existing benchmarks inadequately capture real-world complexity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    A benchmark evaluates LLMs' capabilities in reliable interactive clinical reasoning, designed to assess their reasoning abilities by observing performance on informationally incomplete clinical queries.

    “A benchmark evaluating LLMs' capabilities in reliable interactive clinical reasoning, designed to assess their reasoning abilities by observing performance on informationally incomplete clinical queries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    By comparing the outputs of the three types of tasks, the amount of missing reasoning ability of the model and the degree of help of automatically retrieved knowledge for the model's reasoning can be evaluated.

    “By comparing the outputs of the three types of tasks, the amount of missing reasoning ability of the model and the degree of help of automatically retrieved knowledge for the model's reasoning can be evaluated.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Three types of tasks are set during MedExpQA evaluation: basic input only, basic input plus gold-standard explanation, and basic input plus RAG text.

    “Reasoning: Three types of tasks are set during evaluation: basic input only, basic input plus gold-standard explanation, and basic input plus RAG text.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The real impact of errors varies widely, from harmless inaccuracies to severe consequences.

    “Moreover, the real impact of errors varies widely, from harmless inaccuracies to severe consequences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The six diagnostic reasoning task categories in DR.BENCH comprehensively span the clinical workflow-continuum, designed to evaluate capabilities including medical concept logic, context-aware information retrieval, structured clinical knowledge classification, knowledge-graph-driven causal reasoning, multi-step evidence integration, and knowledge-intensive clinical inference.

    “Reasoning: The six diagnostic reasoning task categories in DR.BENCH comprehensively span the clinical workflow-continuum, designed to evaluate the model's capabilities including: medical concept logic; context-aware information retrieval; structured clinical knowledge classification; knowledge-graph-driven causal reasoning; multi-step evidence integration; knowledge-intensive clinical inference.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Current evaluations remain limited.

    “However, current evaluations remain limited; many essential qualities are difficult to quantify, and existing benchmarks inadequately capture real-world complexity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    The taxonomy includes six core dimensions, each with corresponding assessment methods.

    “The taxonomy includes six core dimensions, each with corresponding assessment methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    In practice, medical QA involves multi-turn conversations, clarifications, follow-up questions, and changing clinical context.

    “In practice, medical QA involves multi-turn conversations, clarifications, follow-up questions, and changing clinical context, dynamics rarely captured by current evaluations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    HELM is not specific to medicine but is often used as a template.

    “Another is Holistic Evaluation of Language Models (HELM) (Liang et al., 2023) -not specific to medicine but often used as a template -which emphasizes transparent reporting of a model's strengths and failures across scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    MedHallBench is a comprehensive medical hallucination evaluation framework integrating automated clinical medical image caption hallucination scoring (ACHMI) and clinical expert review.

    “MedHallBench A comprehensive medical hallucination evaluation framework integrating automated clinical medical image caption hallucination scoring (ACHMI) and clinical expert review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    DR.BENCH is a benchmark for evaluating clinical diagnostic reasoning capabilities of large language models, comprising six reasoning tasks: MedNLI, Assessment and Plan Relation Labeling, EmrQA, SOAP Section Classification, Problem Summarization, and Diagnosis Generation.

    “DR.BENCH A benchmark for evaluating clinical diagnostic reasoning capabilities of large language models (LLMs), comprising six reasoning tasks: MedNLI, Assessment and Plan Relation Labeling, EmrQA, SOAP Section Classification, Problem Summarization, and Diagnosis Generation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    It detects whether the model can correctly classify the labels of question-answer pairs as real or hallucination.

    “Hallucination: Detect whether the model can correctly classify the labels of question-answer pairs as”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    A system that maximizes safety by refusing all borderline queries may sacrifice robustness.

    “A system that maximizes safety by refusing all borderline queries may sacrifice robustness, while one that prioritizes detail could harm explainability or safety.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    It uses lexical metrics and human ratings to score explanation quality.

    “Uses lexical metrics and human ratings to score explanation quality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    A system that prioritizes detail could harm explainability or safety.

    “A system that maximizes safety by refusing all borderline queries may sacrifice robustness, while one that prioritizes detail could harm explainability or safety.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Inter-rater consistency varies.

    “Expert review ensures high-quality critique, but it cannot scale to the volume of queries real systems face, and inter-rater consistency varies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    BiasMedQA is a benchmark dataset for evaluating whether there is bias towards different patient groups such as those of different genders and races in LLMs in medical question answering.

    “BiasMedQA A benchmark dataset for evaluating whether there is bias (towards different patient groups such as those of different genders, races, etc.) in LLMs in medical question answering.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    It tests knowledge in less-studied specialties, which is robustness to specialty domains.

    “Also tests knowledge in less-studied specialties (robustness to specialty domains).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    It evaluates the simulation of a dynamic clinical interaction environment where the model under assessment acts as an Expert System, with performance under informationally incomplete initial conditions recorded to measure interactive clinical reasoning capabilities.

    “Reasoning: Evaluating the simulation of a dynamic clinical interaction environment where the model under assessment acts as an Expert System, with performance under informationally incomplete initial conditions recorded to measure interactive clinical reasoning capabilities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    HELM emphasizes transparent reporting of a model's strengths and failures across scenarios.

    “Another is Holistic Evaluation of Language Models (HELM) (Liang et al., 2023) -not specific to medicine but often used as a template -which emphasizes transparent reporting of a model's strengths and failures across scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    MedExpQA encompasses multiple languages.

    “MedExpQA encompasses multiple languages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The reasoning subset of MedXpertQA comprises highest-difficulty questions requiring multi-step logical reasoning, selected from both Text and MM configurations, specifically designed to evaluate model reasoning capabilities.

    “Reasoning: The reasoning subset comprises highest-difficulty questions requiring multi-step logical reasoning, selected from both Text (text-based) and MM (multimodal) configurations, specifically designed to evaluate model reasoning capabilities”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Incorporating evaluation in the development loop accelerates progress toward trustworthy QA systems suitable for critical medical use.

    “Incorporating evaluation in the development loop accelerates progress toward trustworthy QA systems suitable for critical medical use.”
    Quote in the paperNot a substantial claim
79 of 79

  • The paper relies on this without support.

    In practice, a single evaluation signal often indicates multiple risks, yet this interplay is seldom analyzed or leveraged to guide system development holistically.

    “In practice, a single evaluation signal often indicates multiple risks, yet this interplay is seldom analyzed or leveraged to guide system development holistically.”

  • The paper relies on this without support.

    There is a lack of large, bias-annotated medical QA corpora; most fairness evaluations rely on small, hand-crafted case sets or retrospective analyses of model outputs.

    “However, a major obstacle is the lack of large, bias-annotated medical QA corpora-most evaluations rely on small, hand-crafted case sets or retrospective analyses of model outputs.”

  • The paper relies on this without support.

    Calibration improvements, such as atypicality-aware recalibration reducing ECE by 60%, yield safer behaviour.

    “Thus, calibration improvements (e.g., via atypicality-aware recalibration reducing ECE by 60%) yield safer behaviour.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Umapathi attribution conflicts with the cited authors

    Minor

    The statement attributes the result to Umapathi et al. but cites Savage et al. (2024b); the corresponding reference lists Savage and coauthors, not Umapathi.

  • Alonso study is attributed to the wrong benchmark

    Minor

    The text says Alonso et al. included human annotation in MedExQA, but the cited Alonso paper is titled MedExpQA. The paper separately identifies MedExQA as Kim et al.'s benchmark.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Yella Diekmann, Chase M Fensore, Rodrigo M Carrillo- Larco, Nishant Pradhan, Bhavya Appana, and Joyce C Ho. 2025. Evaluating safety of large lan- guage models for patient-facing medical question an- swering. In Proceedings of the 4th Machine Learning for Health Symposium, volume 259 of Proceedings of Machine Learning Research, pages 267-290.
  2. PMLR.
  3. Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024b. Towards safe large language models for medicine. In ICML 2024 Work- shop on Models of Human Feedback for AI Align- ment.
  4. Jie Huang and Kevin Chen-Chuan Chang. 2023. To- wards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049-1065, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  5. Adam Tauman Kalai and Santosh S. Vempala. 2024. Calibrated language models must hallucinate. New York, NY, USA. Association for Computing Machin- ery. · DOI · Paper page
  6. Karim Lekadir, Alejandro F Frangi, Antonio R Porras, Ben Glocker, Celia Cintas, Curtis P Langlotz, Eva Weicken, Folkert W Asselbergs, Fred Prior, Gary S Collins, Georgios Kaissis, Gianna Tsakou, Irène Bu- vat, Jayashree Kalpathy-Cramer, John Mongan, Ju- lia A Schnabel, Kaisar Kushibar, Katrine Riklund, Kostas Marias, and 30 others. 2025. Future-ai: in- ternational consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ, 388. · DOI · Paper page
  7. Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of the 47th International ACM SI- GIR Conference on Research and Development in Information Retrieval, SIGIR '24, page 2395-2400, New York, NY, USA. Association for Computing Machinery. · DOI
  8. Matthew Walsh, David Schulker, and Shing-hon Lau. 2024. Beyond capable: Accuracy, calibration, and ro- bustness in large language models. Carnegie Mellon University, Software Engineering Institute's Insights (blog). Accessed: 2025-May-16. · DOI

Cited by

Sorted by citations

Related

Authors
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X.
Affiliations
  • Shandong University (CN)
  • Western University (CA)
  • Dalhousie University (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.03659
arXiv
2506.03659
OpenAlex
W4416073431
Semantic Scholar
cf927d5a0044eb56d203335949067c69c8184a45
Cited by
0
References
57
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
medical question answering, trustworthy AI, large language models, factuality, robustness, fairness, safety, explainability, calibration, evaluation benchmark, adversarial fine-tuning, safety alignment, expert evaluation, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review