Large Language Model Performance and Clinical Reasoning Tasks
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and Participants: In this cross-sectional study, performance was evaluated using standardized clinical vignettes from the January 2025 update of MSD Manual vignettes. A total of 21 off-the-shelf LLMs, including recently released GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4, were evaluated. Models were assessed by medical student scorers in triplicate across sequential stages of the standard clinical workflow. Analyses were performed from January to December 2025. Main Outcomes and Measures: The primary outcome was the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, defined as the normalized polygonal area representing balanced accuracy across 5 domains of clinical reasoning as follows: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous clinical reasoning questions. Analyses including analyses of variance, t tests, and regression models were used to compare AI model performance and demographic associations. Results: LLMs were tested across 29 clinical vignettes (representing 16 254 responses in total). PrIME-LLM scores ranged from 0.64 (range, 0.63-0.65) (Gemini 1.5 Flash) to 0.78 (range, 0.77-0.79) (Grok 4), with reasoning-optimized models outperforming nonreasoning models and GPT models scoring highest overall. Differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate. Failure rates exceeded 0.80 (range, 0.90-1.00) for differential diagnosis in all models but were less than 0.40 (range, 0.09-0.39) for final diagnosis. Multimodal performance was robust; most LLM models showed improved accuracy with image inputs. Conclusions and Relevance: In this cross-sectional study of 21 LLMs, frontier LLMs achieved high accuracy on final diagnoses but performed poorly in generating differential diagnoses and navigating uncertainty relative to other reasoning stages. The PrIME-LLM framework provided greater separation than raw accuracy, revealing critical reasoning gaps obscured by traditional benchmarks. Thus, despite version-based improvements and advantages in reasoning-optimized models, off-the-shelf LLMs have not yet achieved the intelligence required for safe deployment and remain limited in demonstrating advanced clinical reasoning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“a pattern mirroring our observation that topics with sparse or uncertain evidence produced the highest hallucination rates (26).”
From Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care (2026) · cites this paper as [26]
Experimental — accuracy not yet measuredIn the citing paper
“Published studies have reported that even state-of-the-art LLMs have failed on differential diagnosis tasks in over 80% of cases, despite achieving high accuracy on final diagnoses”
From Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care (2026) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
Self-citationLeveraging large language models for decision support in personalized oncology
Benary, M., Wang, X. D., Schmidt, M., Soll, D., Hilfenhaus, G., Nassir, M., Sigler, C., Knödler, M., Keller, U., Beule, D., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.43689 · Cited by 306
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 172
The effect of using a large language model to respond to patient messages
Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00060-8 · Cited by 147
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135
Williams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119
Research works printed but not linked: 15
- Koirala P, Thongprayoon C, Miao J,. Evaluating AI performance in nephrology triage and subspecialty referrals. Sci Rep. 2025; 15(1): 3455. doi: 10.1038/s41598-025-88074-5 39870788 PMC11772766 · DOI
- Ong CS, Obey NT, Zheng Y, Cohan A, Schneider EB. SurgeryLLM: a retrieval-augmented generation large language model framework for surgical decision support and workflow enhancement. NPJ Digit Med. 2024; 7(1): 364. doi: 10.1038/s41746-024-01391-3 39695316 PMC11655968 · DOI
- Koranteng E, Rao A, Flores E,. Empathy and equity: key considerations for large language model adoption in health care. JMIR Med Educ. 2023; 9: e51199. doi: 10.2196/51199 38153778 PMC10884892 · DOI
- Rao A, Kim J, Lie W,. Proactive polypharmacy management using large language models: opportunities to enhance geriatric care. J Med Syst. 2024; 48(1): 41. doi: 10.1007/s10916-024-02058-y 38632172 PMC12042286 · DOI · Paper page
- Young CC, Enichen E, Rivera C,. Diagnostic accuracy of a custom large language model on rare pediatric disease case reports. Am J Med Genet A. 2025; 197(2): e63878. doi: 10.1002/ajmg.a.63878 39268988 PMC12123583 · DOI
- Young CC, Enichen E, Rao A, Succi MD. Racial, ethnic, and sex bias in large language model opioid recommendations for pain management. Pain. 2025; 166(3): 511- 517. doi: 10.1097/j.pain.0000000000003388 39283333 PMC12042288 · DOI
- Nguyen D, Rao A, Mazumder A, Succi MD. Exploring the accuracy of embedded ChatGPT-4 and ChatGPT-4o in generating BI-RADS scores: a pilot study in radiologic clinical support. Clin Imaging. 2025; 117: 110335. doi: 10.1016/j.clinimag.2024.110335 39549561 PMC12257487 · DOI
- Kim J, Kincaid JWR, Rao AS,. Risk stratification of potential drug interactions involving common over-the-counter medications and herbal supplements by a large language model. J Am Pharm Assoc (2003). 2025; 65(1): 102304. doi: 10.1016/j.japh.2024.102304 39613295 PMC12042285 · DOI
- Rao A, Mu A, Enichen E,. A future of self-directed patient internet research: large language model-based tools versus standard search engines. Ann Biomed Eng. 2025; 53(5): 1199- 1208. doi: 10.1007/s10439-025-03701-6 40025252 PMC12123582 · DOI
- Succi MD, Chang BS, Rao AS. Building the AI-enabled medical school of the future. JAMA. 2025; 333(19): 1665- 1666. doi: 10.1001/jama.2025.2789 40163081 · DOI
- Rao AS, Kim J, Mu A,. Synthetic medical education in dermatology leveraging generative artificial intelligence. NPJ Digit Med. 2025; 8(1): 247. doi: 10.1038/s41746-025-01650-x 40320492 PMC12050279 · DOI
- Rao AS, Prasad S, Lee RS, Farrell S, McKinley S, Succi MD. Development and evaluation of an artificial intelligence-powered surgical oral examination simulator: a pilot study. Mayo Clin Proc Digit Health. 2025; 3(3): 100241. doi: 10.1016/j.mcpdig.2025.100241 40677929 PMC12270061 · DOI
- Rao AS, Lee RS, Bott E,. The digital standardized patient: an artificial intelligence coach for cultural dexterity in surgical care. J Am Coll Surg. 2025; 241(5): 887- 894. doi: 10.1097/XCS.0000000000001505 40704693 PMC13198390 · DOI
- Shieh A, Tran B, He G, Kumar M, Freed JA, Majety P. Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports. Sci Rep. 2024; 14(1): 9330. doi: 10.1038/s41598-024-58760-x 38654011 PMC11039662 · DOI · Paper page
- Young CC, Enichen E, Rao A,. Pilot study of large language models as an age-appropriate explanatory tool for chronic pediatric conditions. medRxiv. Preprint posted online August 7, 2024. doi: 10.1101/2024.08.06.24311544 · DOI
Cited by
Seifi, A., & Seyfi, A. (2026) · Critical Care Explorations · doi:10.1097/cce.0000000000001474 · Cited by 1
Yano, Y., Kakizaki, H., Nagasu, H., Kishi, S., Koshida, T., Nihei, Y., Hirano, A., Sugawara, Y., Imaizumi, T., Osakabe, Y., et al. (2026) · doi:10.64898/2026.07.15.26358121
Metadata
- Authors
- Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D.
- Affiliations
- Harvard Medical School, Boston, Massachusetts
- Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center, Mass General Brigham, Boston, Massachusetts
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2026-04-13
- Volume
- 9
- Issue
- 4
- Pages
- e264003
- ISSN
- 2574-3805
- PMC
- PMC13077515
- Cited by
- 69
- References
- 12
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.