Holistic evaluation of large language models for medical tasks with MedHELM

May combine two claims.

Benchmark evaluations were done in a private cloud tenant and cost US$805-1,850 per model, which can be prohibitive for resource-limited settings.

“Our benchmark evaluations are done in a private cloud tenant and cost US$805–1,850 per model, which can be prohibitive for resource-limited settings.”

May combine two claims.

DeepSeek R1 and o3-mini lead across most categories, with DeepSeek R1 excelling in clinical note generation (0.85, tied with o3-mini) and patient communication and education (0.89), while o3-mini leads in clinical decision support (0.77) and medical research assistance (0.75).

“At the model level, DeepSeek R1 and o3-mini lead across most categories, with DeepSeek R1 excelling in clinical note generation (0.85, tied with o3-mini) and patient communication and education (0.89), while o3-mini leads in clinical decision support (0.77) and medical research assistance (0.75).”

May combine two claims.

Reasoning models DeepSeek R1 (US$1,850) and o3-mini (US$1,761) incurred higher costs, with the same win rate of 0.66.

“Reasoning models—DeepSeek R1 (US$1,850) and o3-mini (US$1,761)—incurred higher costs, with the same win rate of 0.66.”

May combine two claims.

The LLM-jury achieved an ICC of 0.47 with clinician ratings, outperforming average agreement between clinicians (ICC=0.43) and standard automated metrics ROUGE-L (ICC=0.36) and BERTScore-F (ICC=0.44).

“The LLM-jury achieved an intraclass correlation coefficient (ICC) of 0.47 with clinician ratings, which notably outperformed both the average agreement between clinicians themselves (ICC = 0.43) and standard automated metrics including ROUGE-L (ICC = 0.36) and BERTScore-F (ICC = 0.44; Extended Data Table 4).”

• AssumptionRests on an assumption: Clinician ratings are a valid reference standard for evaluating LLM outputs in medical tasks.
• AssumptionRests on an assumption: The three criteria—accuracy, completeness and clarity—adequately capture the quality of open-ended medical outputs.

May combine two claims.

For moderate-quality responses (rated 3-4), the LLM-jury outperformed ROUGE-L 62.5% of the time and showed stronger correlation with human judgment (0.414 versus 0.386).

“The LLM-jury’s strengths are most pronounced when evaluating moderate-quality responses (rated 3–4 on our five-point scale), where it outperformed ROUGE-L 62.5% of the time and showed stronger correlation with human judgment (0.414 versus 0.386).”

May combine two claims.

The Claude Sonnet series (3.5 and 3.7) demonstrates consistent performance with scores of 0.82-0.83 in clinical note generation and 0.83-0.84 in patient communication and education.

“The Claude Sonnet series (3.5 and 3.7) demonstrates consistent performance with scores of 0.82–0.83 in clinical note generation and 0.83–0.84 in patient communication and education.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Current smaller models face significant limitations on complex clinical tasks.

“In addition, current smaller models face significant limitations on complex clinical tasks; therefore, devising ways of efficient benchmarking with local models on longitudinal datasets remains a fruitful area of future work.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Clinical Decision Support is the most represented category with 12 benchmarks, followed by Patient Communication (8), Clinical Note Generation and Medical Research Assistance (6), and Administration and Workflow (5).

“Clinical Decision Support is the most represented category with 12 benchmarks, followed by Patient Communication (8), Clinical Note Generation and Medical Research Assistance (6), and Administration and Workflow (5).”

May combine two claims.

Gemini models exhibit greater variation, with scores of 0.74-0.78 in clinical note generation and 0.76-0.81 in patient communication and education, where Gemini 2.0 Flash substantially outperforms 1.5 Pro.

“Gemini models exhibit greater variation, with scores of 0.74–0.78 in clinical note generation and 0.76–0.81 in patient communication and education, where Gemini 2.0 Flash substantially outperforms 1.5 Pro.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Access levels are 16 public, 7 gated, and 14 private.

“Access levels are designated as 16 public, 7 gated (that is, requiring approval) and 14 private.”

May combine two claims.

Most models achieve highest scores in clinical note generation (0.74-0.85) and patient communication and education (0.76-0.89), moderate in medical research assistance (0.65-0.75) and clinical decision support (0.63-0.77), and lower in administration and workflow (0.53-0.63).

“Most models achieve their highest scores in clinical note generation (0.74–0.85) and patient communication and education (0.76–0.89), with moderate performance in medical research assistance (0.65–0.75) and clinical decision support (0.63–0.77) and generally lower scores in administration and workflow (0.53–0.63).”

• Also stated in: § Discussion · plain

  “most models perform best in clinical note generation (0.74–0.85) and patient communication and education (0.76–0.89), moderately in medical research assistance (0.65–0.75) and clinical decision support (0.63–0.77), and worst in administration and workflow (0.53–0.63).”

• AssumptionRests on an assumption: The 37 benchmarks are representative of real-world medical tasks.

May combine two claims.

Gemini 2.0 Flash dropped 42 percentile points (86th to 44th) and GPT-4o fell 24 points (80th to 56th) in medical tasks compared to general benchmarks.

“some general-purpose models showed concerning degradation in medical tasks; for example, Gemini 2.0 Flash dropped 42 percentile points (86th to 44th) and GPT-4o fell 24 points (80th to 56th).”

May combine two claims.

The LLM-jury substantially outperformed automated metrics in 32-34% of cases, while automated metrics showed advantages in 27-29% of cases.

“Rather than showing universal superiority, our analysis revealed complementary strengths: the LLM-jury substantially outperformed automated metrics in 32–34% of cases, while automated metrics showed advantages in 27–29% of cases.”

May combine two claims.

The benchmark suite comprises 19 existing benchmarks, 5 reformulated benchmarks, and 13 new benchmarks, of which 12 are EHR based.

“The benchmark suite comprises 19 existing benchmarks, 5 reformulated benchmarks derived from previously unevaluated medical datasets and 13 new benchmarks, of which 12 are EHR based.”

May combine two claims.

Three smaller open-source models (Qwen-2.5-7B-instruct, Phi-3.5-mini-instruct, MedGemma-4b-it) achieved reasonable performance on some text-generation tasks but showed significant deficits in specialized medical reasoning, with poor performance on MedCalc-Bench (0.01-0.091) and Medbullets (0.192-0.406).

“While these models achieved reasonable performance on some text-generation tasks, they showed significant deficits in specialized medical reasoning, with particularly poor performance on medical calculations (MedCalc-Bench: 0.01–0.091) and clinical knowledge assessments (Medbullets: 0.192–0.406).”

May combine two claims.

For clinical documentation tasks, Claude 3.7, DeepSeek R1, and o3-mini excelled at generating coherent clinical narratives, while other models showed limitations.

“For clinical documentation tasks including DischargeMe, ACI-Bench, MIMIC-RRS and NoteExtract, Claude 3.7, DeepSeek R1 and o3-mini excelled at generating coherent clinical narratives, while other models showed limitations in producing contextually appropriate documentation.”

May combine two claims.

Open-source Llama 3.3 Instruct (US$940) had a 0.30 win rate, while Gemini 1.5 Pro (US$1,132) reached 0.23.

“Open-source Llama 3.3 Instruct (US$940) had a 0.30 win rate, while Gemini 1.5 Pro (US$1,132) reached 0.23.”

May combine two claims.

GPT-4o achieved a 58% win rate, Gemini 2.0 Flash 43%, GPT-4o mini 37%, Llama 3.3 Instruct 30%, and Gemini 1.5 Pro 23%.

“GPT-4o achieved a 58% win rate, while Gemini 2.0 Flash (43%) and GPT-4o mini (37%) performed at moderate levels. Open-source Llama 3.3 Instruct achieved a 30% win rate, while Gemini 1.5 Pro ranked lowest with 23% wins”

May combine two claims.

Testing all seven possible three-judge combinations from the evaluation model pool found strong consistency, with average correlations exceeding 0.85 and 100% agreement in identifying top and bottom performing models for 12 of 13 datasets.

“We tested all seven possible three-judge combinations from our pool of evaluation models and found strong consistency across compositions, with average correlations exceeding 0.85 and 100% agreement in identifying the top and bottom performing models for 12 of 13 datasets (Extended Data Table 5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Benchmark distribution across subcategories is uneven, with 15 subcategories containing a single benchmark and 7 containing between 2 and 7.

“The distribution of benchmarks across subcategories is uneven, with 15 subcategories containing a single benchmark and the remaining 7 subcategories containing between 2 and 7 benchmarks each.”

May combine two claims.

Llama 3.3 performs well in patient communication and education (0.81) but shows the lowest score in administration and workflow (0.53).

“The open-source Llama 3.3 performs well in patient communication and education (0.81) but shows the lowest score in administration and workflow (0.53), indicating areas for future improvement”

May combine two claims.

The suite includes 13 open-ended benchmarks and 22 closed-ended benchmarks.

“The suite includes 13 open-ended benchmarks (requiring free-text generation) and 22 closed-ended benchmarks (with predefined answer choices).”

May combine two claims.

Claude 3.7 Sonnet and Claude 3.5 Sonnet achieved win rates of 63% with macro-averages of 0.74 and 0.73.

“Both the Claude 3.7 Sonnet and Claude 3.5 Sonnet models achieved win rates of 63% with similar macro-averages of 0.74 and 0.73, respectively”

May combine two claims.

Claude 3.5 Sonnet (US$1,572) and Claude 3.7 Sonnet (US$1,538) provide a good cost-performance balance, achieving a 0.63 win rate at reduced costs.

“Claude 3.5 Sonnet (US$1,572) and Claude 3.7 Sonnet (US$1,538) provide a good cost–performance balance, achieving a 0.63 win rate at reduced costs.”

May combine two claims.

Non-reasoning models GPT-4o mini (US$805) and Gemini 2.0 Flash (US$815) incurred the lowest costs and achieved win rates of 0.37 and 0.43, respectively.

“As expected, non-reasoning models—GPT-4o mini (US$805) and Gemini 2.0 Flash (US$815)—incurred the lowest costs and achieved win rates of 0.37 and 0.43, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ROUGE and BERTScore were calculated for every open-ended benchmark that had gold-standard responses, but they served only as secondary metrics because their alignment with clinical judgment is limited.

“ROUGE and BERTScore were also computed for all open-ended benchmarks with gold-standard responses but used only as secondary metrics due to their limited alignment with clinicaljudgment.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claude 3.7 Sonnet had mean accuracy 3.910, completeness 3.388, and clarity/structure 4.744 as a judge.

“Claude 3.7 Sonnet | 3.910 | 0.884 | 3.388 | 1.204 | 4.744 | 0.457”

May combine two claims.

GPT-4o ranked 5th of 9 in MedHELM, 45th of 224 in LM Arena, and 25th of 51 in HELM, with a medical performance drop of −24 to +5 points.

“GPT-4o | 56th (5th of 9) | 80th (45th of 224) | 51st (25th of 51) | Medical: −24 to +5 points drop”

The quote only partly states this claim.

Resource-efficient models perform worse than frontier LLMs across the medical tasks examined.

“Results show the performance gap between resource-efficient models and frontier LLMs across medical tasks.”

May combine two claims.

The refined taxonomy consists of 5 categories, 22 subcategories, and 121 tasks.

“Based on the comments, we refined definitions and expanded the taxonomy to have 5 categories, 22 subcategories and 121 tasks.”

May combine two claims.

On MEDIQA, LLM-jury agreement was 0.625, ROUGE-L 0.343, BERTScore-F 0.668, and clinician-clinician 0.520.

“MEDIQA | 0.625 (0.150, 0.830) | 0.343 (0, 0.710) | 0.668 (0.250, 0.850) | 0.520 (0.500, 0.534)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MedHELM comprises 37 benchmarks, grouped by clinical category, access level, and curation status.

“Overview of all 37 benchmarks included in MedHELM, categorized by category, access level and curation status”

May combine two claims.

Gemini 2.0 Flash ranked 6th of 9 in MedHELM, 31st of 224 in LM Arena, and 18th of 51 in HELM, with a medical performance drop of −21 to −42 points.

“Gemini 2.0 Flash | 44th (6th of 9) | 86th (31st of 224) | 65th (18th of 51) | Medical: −21 to −42 points drop”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-4o had mean accuracy 4.355, completeness 3.816, and clarity/structure 4.892 as a judge.

“GPT-4o | 4.355 | 0.648 | 3.816 | 0.909 | 4.892 | 0.310”

May combine two claims.

o3-mini ranked 2nd of 9 in MedHELM, 43rd of 224 in LM Arena, and was not ranked in HELM, with a medical performance gain of +8 points.

“o3-mini | 89th (2nd of9) | 81st (43rd of 224) | Not ranked | Medical: +8 points gain”

May combine two claims.

Claude 3.7 Sonnet ranked 3rd of 9 in MedHELM, 29th of 224 in LM Arena, and 19th of 51 in HELM, with a medical performance shift of −9 to +15 points.

“Claude 3.7 Sonnet | 78th (3rd of 9) | 87th (29th of 224) | 63rd (19th of 51) | Medical: −9 to +15 points shift”

May combine two claims.

DeepSeek R1 ranked 1st of 9 in MedHELM, 16th of 224 in LM Arena, and 16th of 51 in HELM, with a medical performance gain of +7-27 points.

“DeepSeek R1 | 100th (1st of 9) | 93rd (16th of 224) | 73rd (16th of 51) | Medical: +7–27 points gain”

May combine two claims.

Administration and workflow was underrepresented in existing benchmarks, motivating the curation of new private datasets.

“To address the underrepresentation of ‘administration and workflow’, we partnered with Stanford Healthcare to curate private datasets for tasks that are routinely done in health systems but for which benchmark datasets do not exist (for example, referral triage, scheduling).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The estimated expense for refreshing the MedHELM leaderboard once, covering nine models and 37 benchmarks, is $11,825.97.

“For nine models and 37 benchmarks in the current MedHELM suite, one update to the leaderboard is estimated to cost 11, 825.97.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLaMA 3.3 70B had mean accuracy 4.250, completeness 3.785, and clarity/structure 4.969 as a judge.

“LLaMA 3.3 70B | 4.250 | 0.613 | 3.785 | 0.864 | 4.969 | 0.218”

May combine two claims.

Claude 3.5 Sonnet ranked 4th of 9 in MedHELM, 30th of 224 in LM Arena, and 24th of 51 in HELM, with a medical performance shift of −20 to +14 points.

“Claude 3.5 Sonnet | 67th (4th of 9) | 87th (30th of 224) | 53rd (24th of 51) | Medical: −20 to +14 points shift”

May combine two claims.

On the combined open-ended benchmarks, LLM-jury agreement with clinician ratings was ICC 0.474, compared with 0.361 for ROUGE-L, 0.441 for BERTScore-F, and 0.426 for average clinician-clinician agreement.

“Combined | 0.474 (0.100, 0.690) | 0.361 (0, 0.630) | 0.441 (0.050, 0.670) | 0.426 (0.295, 0.585)”

May combine two claims.

On ACI-Bench, LLM-jury agreement was 0.305, ROUGE-L 0.445, BERTScore-F 0.250, and clinician-clinician 0.458.

“ACI-Bench | 0.305 (0, 0.670) | 0.445 (0, 0.730) | 0.250 (0, 0.640) | 0.458 (0.201, 0.945)”

May combine two claims.

Medical licensing exams and diagnostic tasks are prioritized in 64% of LLM healthcare evaluations, while many administrative and operational tasks are often overlooked.

“While medical licensing exams and diagnostic tasks are prioritized in 64% of LLM healthcare evaluations8, many administrative and operational tasks that represent significant human bottlenecks are often overlooked.”

• cites [8] Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333, 319–328 (2025). sourceNot checked yet

  Not checked in the free run.

• cites [9] Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613–2622 (2024). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

HealthBench's design does not capture iterative, context-aware interactions expected from an assistive copilot, nor does it assess structured tasks that dominate everyday workflows.

“However, this design does not capture the iterative, context-aware interactions clinicians expect from an assistive copilot, nor does it assess performance on structured tasks that dominate everyday workflows (for example, order review, note generation, literature summarization).”

• cites [10] Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf (2025).Not checked yet

  Not checked in the free run.

May combine two claims.

Existing benchmarks use synthetic vignettes or narrowly scoped exam questions and fail to capture key aspects of real diagnostic processes, such as extracting relevant details from patient records.

“Existing benchmarks rely on synthetic vignettes or narrowly scoped exam questions, failing to capture key aspects of real diagnostic processes such as extracting relevant details from patient records6,7.”

• cites [6] Raji, I. D., Daneshjou, R. & Alsentzer, E. It’s time to bench the medical exam benchmark. NEJM AI 2, AIe2401235 (2025). sourceNot checked yet

  Not checked in the free run.

• cites [7] Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning 174, 248–260 (PMLR, 2022).Not checked yet

  Not checked in the free run.

May combine two claims.

HealthBench's physician-authored rubrics reward exhaustive, risk-averse responses that maximize safety and completeness, offering a stress test for fully autonomous chatbots.

“Its physician-authored rubrics reward exhaustive, risk-averse responses that maximize safety and completeness, offering a valuable stress test for fully autonomous chatbots.”

• cites [10] Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf (2025).Not checked yet

  Not checked in the free run.

May combine two claims.

The LLM-jury approach, by beating clinician-clinician agreement, enables scalable evaluation of open-ended model outputs without requiring extensive clinician time.

“By beating clinician–clinician agreement, this approach enables scalable evaluation of open-ended model outputs without requiring extensive clinician time, a scarce and expensive resource.”

May combine two claims.

Although Phi-3.5-mini-instruct can fit on such hardware, the memory requirements for the benchmark's long patient timelines exceeded available memory.

“Although models such as Phi-3.5-mini-instruct can fit on such hardware, the memory requirements for the benchmark’s long patient timelines exceeded available memory.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The higher performance in communication tasks than administrative ones may stem from administrative workflows using data not seen during training.

“The higher performance in communication tasks than administrative ones may stem from administrative workflows using data not seen during training, warranting caution in healthcare AI implementation for back-office tasks without quantifying task-specific performance.”

May combine two claims.

Most LLM evaluations in medicine still rely on closed-form QA over exam-style datasets such as MEDQA and MEDMCQA, with only ~5% incorporating real EHR data and very few addressing free-text-generation tasks or cost-aware metrics.

“Most LLM evaluations in medicine still rely on closed-form QA over exam-style datasets such as MEDQA and MEDMCQA, with only ~5% incorporating real EHR data and very few addressing free-text-generation tasks or cost-aware metrics7,16,17.”

• cites [7] Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning 174, 248–260 (PMLR, 2022).Not checked yet

  Not checked in the free run.

• cites [16] Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021). sourceNot checked yet

  Not checked in the free run.

• cites [17] Wornow, M. et al. Context clues: evaluating long context models for clinical prediction tasks on EHRs. In Proc. 13th International Conference on Learning Representations https://openreview.net/pdf?id=zg3ec1TdAP (ICLR, 2025).Not checked yet

  Not checked in the free run.

May combine two claims.

MedHELM organizes tasks by clinical function rather than NLP task type, incorporates real-world EHR data, and includes cost-performance analysis absent from BRIDGE and BigBIO.

“In contrast, MedHELM organizes tasks by clinical function rather than NLP task type (for example, NLI, NER), incorporates real-world EHR data and includes cost–performance analysis absent from these frameworks.”

May combine two claims.

The LLM-jury's ICC of 0.47 matched the level of agreement observed between human medical evaluators, reflecting inherent subjectivity in clinical assessment.

“While an ICC of 0.47 may appear modest, it matched the level of agreement observed between human medical evaluators, reflecting the inherent subjectivity in clinical assessment.”

• AssumptionRests on an assumption: Clinician ratings are a valid reference standard for evaluating LLM outputs in medical tasks.

May combine two claims.

Several MedHELM benchmarks include reported estimates of human performance from their original publications, but these estimates are not directly comparable due to differing protocols.

“Several MedHELM benchmarks include reported estimates of human performance from their original publications, serving as reference points for interpreting model results. Because human evaluation protocols differ across studies in task definitions, participant expertise and scoring methods, these estimates are not directly comparable”

May combine two claims.

Traditional automated metrics like ROUGE-L and BERTScore only measure lexical similarity and cannot assess medical accuracy, clinical appropriateness, and safety considerations.

“Traditional automated metrics like ROUGE-L and BERTScore only measure lexical similarity between model outputs and reference answers, but cannot assess the medical accuracy, clinical appropriateness and safety considerations that are critical for healthcare applications.”

May combine two claims.

In a resource-constrained setting, models such as Claude 3.5 Sonnet offer a balance, achieving a win rate of 0.63 at a lower cost.

“In a resource-constrained setting, models such as Claude 3.5 Sonnet offer a balance, achieving a win rate of 0.63 at a lower cost.”

May combine two claims.

The cost-performance trade-off shows that while reasoning models have superior performance, their substantially higher costs may not justify their deployment for all tasks.

“The cost–performance trade-off shows that while reasoning models have superior performance, their substantially higher costs may not justify their deployment for all tasks.”

May combine two claims.

The design isolates model-to-model differences and identifies the best-performing off-the-shelf systems.

“This design isolates model-to-model differences and identifies the best-performing off-the-shelf systems, providing a necessary foundation for subsequent work that will evaluate LLMs in augmentative settings with human experts or against specialized ML models.”

May combine two claims.

Performance variations reflect distinct task-based challenges: free-text generation leverages models' natural language strengths, while structured reasoning tasks require domain-specific knowledge integration and logical inference.

“These performance variations reflect distinct task-based challenges: free-text-generation tasks leverage models’ natural language strengths, while structured reasoning tasks require domain-specific knowledge integration and logical inference.”

May combine two claims.

These frameworks typically focus on narrow clinical scenarios (1-28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.

“However, these frameworks typically focus on narrow clinical scenarios (1–28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.”

• cites [19] Wu, C. et al. Towards evaluating and building versatile large language models for medicine. NPJ Digit. Med. 8, 58 (2025). sourceNot checked yet

  Not checked in the free run.

• cites [20] Ouyang, Z. et al. CliMedBench: a large-scale Chinese benchmark for evaluating medical large language models in clinical scenarios. In Proc. 2024 Conference on Empircal Methods in Natural Language Processing 8428–8438 (EMNLP, 2024). sourceNot checked yet

  Not checked in the free run.

• cites [21] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [22] Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31, 2546–2549 (2025). sourceNot checked yet

  Not checked in the free run.

• cites [23] Cai, Y. et al. MedBench: a large-scale Chinese benchmark for evaluating medical large language models. In Proc. 38th AAAI Conference on Artificial Intelligence 38, 17709–17717 (AAAI, 2024). sourceNot checked yet

  Not checked in the free run.

• cites [24] Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. Conference on Computational Natural Language Learning (CoNLL) 314–334 (CoNLL, 2023). sourceNot checked yet

  Not checked in the free run.

• cites [25] Han, T., Kumar, A., Agarwal, C. & Lakkaraju, H. MedSafetyBench: evaluating and improving the medical safety of large language models. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks https://openreview.net/pdf?id=cFyagd2Yh4 (2024).Not checked yet

  Not checked in the free run.

• cites [26] Liu, F. et al. Application of large language models in medicine. Nat. Rev. Bioeng. 3, 85–104 (2025). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Recent medical benchmarking efforts have begun addressing some limitations: HEALTHBENCH incorporates physician-developed rubrics, ClinicBench advances multimodal evaluation, and automated frameworks explore LLM-as-a-Judge strategies for EHR summarization.

“Recent medical benchmarking efforts (Extended Data Table 8) have begun addressing some limitations: HEALTHBENCH10 incorporates physician-developed rubrics for health conversations, ClinicBench18,19 advances multimodal evaluation, and automated frameworks explore LLM-as-a-Judge strategies for EHR summarization19–26.”

• cites [10] Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf (2025).Not checked yet

  Not checked in the free run.

• cites [18] Liu, F. et al. Large language models in the clinic: a comprehensive benchmark. Preprint at https://arxiv.org/abs/2405.00716 (2024).Not checked yet

  Not checked in the free run.

• cites [19] Wu, C. et al. Towards evaluating and building versatile large language models for medicine. NPJ Digit. Med. 8, 58 (2025). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The LLM-jury evaluates medical content, clinical reasoning, and safety implications, unlike conventional automated metrics that only capture surface-level text similarity.

“Importantly, unlike conventional automated metrics that only capture surface-level text similarity, our LLM-jury evaluates the medical content, clinical reasoning and safety implications that matter most for healthcare applications.”

• AssumptionRests on an assumption: The three criteria—accuracy, completeness and clarity—adequately capture the quality of open-ended medical outputs.

May combine two claims.

For clinical knowledge support, GPT-4o, Claude 3.5, DeepSeek R1, and o3-mini showed high performance, suggesting traditional QA-style questions do not fully capture model capability differences.

“The domain of providing clinical knowledge support, covering fact-intensive tasks such as HeadQA, Medbullets, MedQA and MedMCQA, highlighted performance of GPT-4o, Claude 3.5, DeepSeek R1 and o3-mini, suggesting that traditional question–answering (QA)-style questions do not fully capture model capability differences.”

Classified as probably not a substantial claim. May combine two claims.

The survey assessed both categorical organization and real-world relevance.

“The survey assessed both categorical organization and real-world relevance.”

Classified as probably not a substantial claim.

All jury prompts use a standardized structure.

“All jury prompts follow a standardized structure to ensure consistent evaluation across benchmarks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The two benchmarks were selected for being publicly available and for covering distinct categories: patient communication and education, and clinical note generation.

“We selected these benchmarks because they were publicly available (facilitating annotation) and represent different categories (patient communication and education, and clinical note generation, respectively).”

• AssumptionRests on an assumption: The selected benchmarks MEDIQA and ACI-Bench are representative of open-ended medical tasks for validating LLM-jury.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the first section, clinicians assigned each of 21 subcategories to one of 5 main categories.

“In the first section, clinicians assigned each of our 21 subcategories to one of the 5 main categories.”

Classified as probably not a substantial claim. May combine two claims.

Two intraclass correlations were used to assess LLM-jury-clinician agreement, following z-score normalization of each rater's composite.

“We assessed agreement between the LLM-jury and clinicians via two ICCs, after applying z-score normalization to each rater’s composite (mean of the three axis ratings) to remove scale biases according to equation (1)”

• AssumptionRests on an assumption: Z-score normalization of each rater’s composite removes scale biases.

Probably not a substantial claim: may not pass S3 (load-bearing).

For LLM-jury evaluation, all three jury members use identical task-specific prompts.

“For LLM-jury evaluation, all three jury members use identical task-specific prompts (Supplementary Fig. 2).”

May combine two claims. The quote only partly states this claim.

The taxonomy was systematically validated by clinicians who rated its comprehensiveness on a five-point scale and provided open-ended feedback, assessing both its organizational logic and its coverage of routine medical tasks.

“In the second section, clinicians evaluated the comprehensiveness of our taxonomy on a five-point scale, where a score of 5 means that our categorization covered all routine medical tasks and a score of 1 means it covered very few.”

• Also stated in: § Development of the taxonomy of tasks · plain

  “This systematic validation approach evaluated both the taxonomy’s organizational logic and its comprehensiveness in representing actual medical tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Advanced LLM-as-Jury systems use chain-of-thought reasoning for each evaluator and specialized tools to detect hallucinations and factual errors.

“Advanced implementations incorporate chain-of-thought reasoning for each evaluator and specialized tools for detecting hallucinations and factual errors30,31.”

• cites [30] Madaan, L. et al. Quantifying variance in evaluation benchmarks. Preprint at https://arxiv.org/abs/2406.10229 (2024).Not checked yet

  Not checked in the free run.

• cites [31] Manakul, P., Liusie, A. & Gales, M. J. F. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proc. 2023 Conference on Empirical Methods Natural Language Processing 9004–9017 (EMNLP, 2023). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A three-member jury was selected based on prior research on odd-numbered panels.

“We selected a three-member jury based on prior research demonstrating that odd-numbered panels reduce tie scenarios while maintaining reliability32.”

• cites [32] Guha, B. Secret ballots and costly information gathering: the jury size problem revisited. MPRA Paper no. 73048 (2016).Not checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Traditional automated metrics such as ROUGE and BLEU are inadequate for assessing medical content quality.
• AssumptionRests on an assumption: Single LLM judges suffer from high variance and systematic biases.

Classified as probably not a substantial claim. May combine two claims.

The two agreement metrics together measure the fidelity of LLM-jury to expert clinician rating and whether LLM-clinician alignment approaches inter-clinician agreement.

“Together, these metrics measure (i) the fidelity of LLM-jury to expert clinician rating and (ii) whether LLM-Clinician alignment approaches inter-clinician agreement.”

• AssumptionRests on an assumption: Clinician ratings provide a valid reference standard for evaluating open-ended medical task responses.

Classified as probably not a substantial claim. May combine two claims.

Twenty clinicians, from various specialties, each scored some responses on the same three axes; every instance had at least two clinician raters.

“Twenty clinicians across various specialties each scored a subset of responses on the same three axes, with at least two clinicians per instance.”

May combine two claims.

Early attempts at creating a taxonomy of tasks in medical settings either harmonize datasets into broad categories without clinician input or collapse heterogeneous skills under a single 'generation' label.

“Early attempts (BIGBIO, ClinicBench) at creating a taxonomy of tasks in medical settings either harmonize datasets into broad categories without clinician input or collapse heterogeneous skills under a single ‘generation’ label14,18.”

• cites [14] Fries, J. A. et al. BigBio: a framework for data-centric biomedical natural language processing. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks https://openreview.net/pdf?id=8lQDn9zTQlW (2022).Not checked yet

  Not checked in the free run.

• cites [18] Liu, F. et al. Large language models in the clinic: a comprehensive benchmark. Preprint at https://arxiv.org/abs/2405.00716 (2024).Not checked yet

  Not checked in the free run.

May combine two claims.

The jury comprised GPT-4o, Claude 3.7 Sonnet, and LLaMA 3.3 70B to represent diverse architectures and minimize provider-specific bias.

“The jury composition (GPT-4o, Claude 3.7 Sonnet, LLaMA 3.3 70B) was chosen to represent diverse model architectures and training approaches, minimizing systematic bias from any single provider.”

• AssumptionRests on an assumption: Traditional automated metrics such as ROUGE and BLEU are inadequate for assessing medical content quality.
• AssumptionRests on an assumption: Single LLM judges suffer from high variance and systematic biases.

May combine two claims.

Among the compared frameworks, MedHELM has the widest coverage: 37 datasets spanning all five clinical categories, real patient data from several sources, and several evaluation dimensions.

“MedHELM covers the broadest scope with 37 datasets across all five clinical categories, real patient data from multiple sources, and multiple dimensions of evaluation, where the dimensions are taken from [MA8].”

• cites [MA8]Not checked yet

  Not checked in the free run.

May combine two claims.

MedHELM uses a three-judge ensemble without chain-of-thought prompting to balance evaluation reliability and computational efficiency.

“MedHELM implements a three-judge ensemble without chain-of-thought prompting to optimize the balance between evaluation reliability and computational efficiency.”

• AssumptionRests on an assumption: Traditional automated metrics such as ROUGE and BLEU are inadequate for assessing medical content quality.
• AssumptionRests on an assumption: Single LLM judges suffer from high variance and systematic biases.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study examined average correlation, consistency in top ranks, and consistency in bottom ranks over all seven jury-composition permutations.

“Analysis of average correlation, first place consistency, and last place consistency across all 7 permutations of jury compositions”

Classified as probably not a substantial claim. May combine two claims.

For each clinician pair sharing at least two instances, composites were z-normalized and ICC(3,k=2) calculated; the pairwise ICCs were then averaged, with a bootstrapped 95% CI, to provide an inter-clinician agreement baseline.

“For every pair of clinicians who scored at least two common instances, we z-normalize each rater’s composite scores and compute ICC(3, k = 2). We then average these pairwise ICCs (and bootstrap a 95% confidence interval) to give a representative inter-clinician agreement baseline.”

• AssumptionRests on an assumption: Z-score normalization of each rater’s composite removes scale biases.

May combine two claims.

All prompts are task specific, not model specific; each benchmark uses a single standardized prompt applied uniformly across all models, ensuring fair comparison and scalability.

“All prompts are task specific, not model specific. Each benchmark uses a single standardized prompt applied uniformly across all models, ensuring fair comparison and scalability.”

Classified as probably not a substantial claim.

For every benchmark, minimum detectable effect values were derived under a paired evaluation framework.

“To ensure adequate statistical power for detecting meaningful performance differences, we calculated minimum detectable effect (MDE) values for each benchmark using a paired evaluation framework.”

• AssumptionRests on an assumption: The MDE framework with α = 0.05 and β = 0.20 ensures adequate statistical power for detecting meaningful performance differences.

Probably not a substantial claim: may not pass S3 (load-bearing).

This exercise tested whether the taxonomy structure matched how clinicians naturally organize medical tasks.

“This exercise tested whether our taxonomy structure matched how clinicians naturally organize medical tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Private datasets were excluded from the smaller open-source model evaluation due to infrastructure limitations in PHI-compliant environments.

“Private datasets were excluded due to infrastructure limitations in PHI-compliant environments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All experiments were run on a shared cluster compliant with protected health information rules and HIPAA.

“All experiments were conducted on a PHI-compliant shared cluster maintaining full Health Insurance Portability and Accountability Act (HIPAA) compliance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All seven possible jury combinations were evaluated to ensure robustness.

“To ensure robustness, we evaluated all seven possible jury combinations; this robustness analysis is detailed in the ‘Results’ (Extended Data Tables 5 and 6).”

• AssumptionRests on an assumption: Traditional automated metrics such as ROUGE and BLEU are inadequate for assessing medical content quality.
• AssumptionRests on an assumption: Single LLM judges suffer from high variance and systematic biases.

May combine two claims.

The extensible framework enables healthcare systems to contribute additional private benchmarks, creating a distributed evaluation ecosystem that provides comprehensive assessment coverage while preventing benchmark gaming by model developers.

“The extensible framework enables healthcare systems to contribute additional private benchmarks, creating a distributed evaluation ecosystem that provides comprehensive assessment coverage while preventing benchmark gaming by model developers.”

May combine two claims.

Previous efforts typically focus on 1-3 categories, while MedHELM uniquely spans all 5 healthcare domains, with administration and workflow representing a previously unaddressed critical gap.

“While previous efforts typically focus on 1–3 categories, MedHELM uniquely spans all 5 healthcare domains, with administration and workflow representing a previously unaddressed critical gap in medical AI evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Total cost is an estimated per-model expense and an upper bound that assumes maximum output token use.

“The total cost reflects the estimated per-model expense of running a MedHELM evaluation across 37 benchmarks and represents an upper bound based on maximum output token usage.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The evaluated instances per dataset are a subset used for model evaluation; larger datasets were capped near 1,000 because of computational limits.

“Instances evaluated represents the subset of each dataset used for model evaluation, capped at approximately 1,000 instances for larger datasets due to computational constraints.”

• AssumptionRests on an assumption: The 1,000-instance cap does not materially affect benchmark comparisons.

Classified as probably not a substantial claim.

Nine state-of-the-art large language models were evaluated using a uniform prompting and decoding protocol.

“We evaluated nine state-of-the-art LLMs under a uniform prompting and decoding regimen.”