It’s Time to Bench the Medical Exam Benchmark
Abstract
AbstractMedical licensing examinations, such as the United States Medical Licensing Examination, have become the default benchmarks for evaluating large language models (LLMs) in health care. Performance on these benchmarks is frequently cited as evidence of progress and used to justify the deployment of LLMs into clinical settings. However, we argue that these benchmarks are fundamentally limited as signals for assessing true clinical utility.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Risk classification and board-style diagnostic reasoning are poor proxies for the open-ended, ambiguous, multi-turn help-seeking that likely constitutes most real usage.
“and risk classification and board-style diagnostic reasoning are poor proxies for the open-ended, ambiguous, multiturn help-seeking that likely constitutes most real usage (Raji et al., 2025;Stade et al., 2024;Luo et al., 2025).”
From HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench (2026) · cites this paper as Raji et al., 2025
Existing benchmarks use synthetic vignettes or narrowly scoped exam questions and fail to capture key aspects of real diagnostic processes, such as extracting relevant details from patient records.
“Existing benchmarks rely on synthetic vignettes or narrowly scoped exam questions, failing to capture key aspects of real diagnostic processes such as extracting relevant details from patient records6,7.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [6]
Cited by
Sorted by citations
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.266 · Cited by 13 · see citing works
One shot at trust: building credible evidence for medical artificial intelligence
Kouzy, R., Hong, J. C., & Bitterman, D. S. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100883 · Cited by 4 · see citing works
Auditing medical multi-agent AI reveals risks of false consensus
Zhu, Y., Gu, L., Wang, Z., Sang, H., Sui, D., Tang, W., Mi, L., Wang, Y., Gao, J., Yao, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.10185 · Cited by 2 · see citing works
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 1 · see citing works
“29 Moreover, such benchmarks harbor inherent vulnerabilities, as LLMs may sometimes arrive at the correct answer through flawed reasoning pathways.”
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Flathers, M., Nguyen, P. A., Noorily, J., Herpertz, J., Chen, M., Multani, J., Powell, S., Granof, M., Kalinch, M., & Torous, J. (2026) · arXiv · doi:10.48550/arxiv.2608.25071
“…detection and diagnostic classification (Flathers et al., 2026), and risk classification and board-style diagnostic reasoning are poor proxies for the open-ended, ambiguous, multi-turn help-seeking that likely constitutes most real usage (Raji et al., 2025; Stade et al., 2024; Luo et al., 2025).”
- Authors
- Raji, I. D., Daneshjou, R., & Alsentzer, E.
- Affiliations
- University of California, Berkeley (US)
- Stanford University (US)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-01-23
- Volume
- 2
- Issue
- 2
- Pages
- AIe2401235-AIe2401235
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4406740982
- Cited by
- 42
- References
- 0
- Citations by year
- 2026: 20 · 2025: 22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Healthcare cost, quality, practices
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- medical licensing examinations, medical licensing, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
