CoChat

Large Language Model Performance and Clinical Reasoning Tasks

Rao, Arya S., Esmail, Kaiz P., Lee, Richard S., Jiang, Sharon, Arraiza Carlo, Bianca, Gill, Jasleen, Khanna, Praneet, Kalmowitz, Ezra, Montagnese, Basile, Heydari, Kimia, Jiao, Qiao, Bott, Ethan, Nguyen, Dan, Wang, Grace, Hood, Michael, Landman, Adam B., Succi, Marc D. · 2026 · JAMA Network Open
Cited by 69

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and Participants: In this cross-sectional study, performance was evaluated using standardized clinical vignettes from the January 2025 update of MSD Manual vignettes. A total of 21 off-the-shelf LLMs, including recently released GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4, were evaluated. Models were assessed by medical student scorers in triplicate across sequential stages of the standard clinical workflow. Analyses were performed from January to December 2025. Main Outcomes and Measures: The primary outcome was the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, defined as the normalized polygonal area representing balanced accuracy across 5 domains of clinical reasoning as follows: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous clinical reasoning questions. Analyses including analyses of variance, t tests, and regression models were used to compare AI model performance and demographic associations. Results: LLMs were tested across 29 clinical vignettes (representing 16 254 responses in total). PrIME-LLM scores ranged from 0.64 (range, 0.63-0.65) (Gemini 1.5 Flash) to 0.78 (range, 0.77-0.79) (Grok 4), with reasoning-optimized models outperforming nonreasoning models and GPT models scoring highest overall. Differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate. Failure rates exceeded 0.80 (range, 0.90-1.00) for differential diagnosis in all models but were less than 0.40 (range, 0.09-0.39) for final diagnosis. Multimodal performance was robust; most LLM models showed improved accuracy with image inputs. Conclusions and Relevance: In this cross-sectional study of 21 LLMs, frontier LLMs achieved high accuracy on final diagnoses but performed poorly in generating differential diagnoses and navigating uncertainty relative to other reasoning stages. The PrIME-LLM framework provided greater separation than raw accuracy, revealing critical reasoning gaps obscured by traditional benchmarks. Thus, despite version-based improvements and advantages in reasoning-optimized models, off-the-shelf LLMs have not yet achieved the intelligence required for safe deployment and remain limited in demonstrating advanced clinical reasoning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2026.4003

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores and Comparison of PrIME-LLM Scores and Overall Accuracy for Tested Clinical Vignettes A, Radar plots show accuracy across 5 domains of clinical reasoning: differential diagnosis (DD), diagnostic testing (DT), final diagnosis (FD), management (M), and miscellaneous clinical reasoning questions (Q). PrIME-LLM scores were calculated as the normalized polygonal area, summarizing balanced performance across tasks. B, Bar graphs show a comparison of overall accuracy vs PrIME-LLM scores for the tested clinical vignettes. Error bars indicate SEMs.
Figure 1. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores and Comparison of PrIME-LLM Scores and Overall Accuracy for Tested Clinical Vignettes A, Radar plots show accuracy across 5 domains of clinical reasoning: differential diagnosis (DD), diagnostic testing (DT), final diagnosis (FD), management (M), and miscellaneous clinical reasoning questions (Q). PrIME-LLM scores were calculated as the normalized polygonal area, summarizing balanced performance across tasks. B, Bar graphs show a comparison of overall accuracy vs PrIME-LLM scores for the tested clinical vignettes. Error bars indicate SEMs.CC BY · PubMed Central
Figure 2. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores by Model Family and PrIME-LLM Scores by Release Date A, Bar graph shows mean PrIME-LLM scores grouped by developer family (GPT, Claude, DeepSeek, Gemini, and Grok). Error bars indicate SEMs. B, Scatter plot shows version-based trajectories within each model family, showing successive releases in chronological order. ANOVA indicates analysis of variance.
Figure 2. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores by Model Family and PrIME-LLM Scores by Release Date A, Bar graph shows mean PrIME-LLM scores grouped by developer family (GPT, Claude, DeepSeek, Gemini, and Grok). Error bars indicate SEMs. B, Scatter plot shows version-based trajectories within each model family, showing successive releases in chronological order. ANOVA indicates analysis of variance.CC BY · PubMed Central
Figure 3. Bar Graphs of Failure Rates of Tested Large Language Models A, Failure rate by age group, defined as the proportion of questions not answered fully correctly. B, Failure rate by patient sex. DD indicates differential diagnosis; DT, diagnostic testing; FD, final diagnosis; M, management; Q, miscellaneous clinical reasoning questions.
Figure 3. Bar Graphs of Failure Rates of Tested Large Language Models A, Failure rate by age group, defined as the proportion of questions not answered fully correctly. B, Failure rate by patient sex. DD indicates differential diagnosis; DT, diagnostic testing; FD, final diagnosis; M, management; Q, miscellaneous clinical reasoning questions.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 15

  1. Koirala P, Thongprayoon C, Miao J,. Evaluating AI performance in nephrology triage and subspecialty referrals. Sci Rep. 2025; 15(1): 3455. doi: 10.1038/s41598-025-88074-5 39870788 PMC11772766 · DOI
  2. Ong CS, Obey NT, Zheng Y, Cohan A, Schneider EB. SurgeryLLM: a retrieval-augmented generation large language model framework for surgical decision support and workflow enhancement. NPJ Digit Med. 2024; 7(1): 364. doi: 10.1038/s41746-024-01391-3 39695316 PMC11655968 · DOI
  3. Koranteng E, Rao A, Flores E,. Empathy and equity: key considerations for large language model adoption in health care. JMIR Med Educ. 2023; 9: e51199. doi: 10.2196/51199 38153778 PMC10884892 · DOI
  4. Rao A, Kim J, Lie W,. Proactive polypharmacy management using large language models: opportunities to enhance geriatric care. J Med Syst. 2024; 48(1): 41. doi: 10.1007/s10916-024-02058-y 38632172 PMC12042286 · DOI · Paper page
  5. Young CC, Enichen E, Rivera C,. Diagnostic accuracy of a custom large language model on rare pediatric disease case reports. Am J Med Genet A. 2025; 197(2): e63878. doi: 10.1002/ajmg.a.63878 39268988 PMC12123583 · DOI
  6. Young CC, Enichen E, Rao A, Succi MD. Racial, ethnic, and sex bias in large language model opioid recommendations for pain management. Pain. 2025; 166(3): 511- 517. doi: 10.1097/j.pain.0000000000003388 39283333 PMC12042288 · DOI
  7. Nguyen D, Rao A, Mazumder A, Succi MD. Exploring the accuracy of embedded ChatGPT-4 and ChatGPT-4o in generating BI-RADS scores: a pilot study in radiologic clinical support. Clin Imaging. 2025; 117: 110335. doi: 10.1016/j.clinimag.2024.110335 39549561 PMC12257487 · DOI
  8. Kim J, Kincaid JWR, Rao AS,. Risk stratification of potential drug interactions involving common over-the-counter medications and herbal supplements by a large language model. J Am Pharm Assoc (2003). 2025; 65(1): 102304. doi: 10.1016/j.japh.2024.102304 39613295 PMC12042285 · DOI
  9. Rao A, Mu A, Enichen E,. A future of self-directed patient internet research: large language model-based tools versus standard search engines. Ann Biomed Eng. 2025; 53(5): 1199- 1208. doi: 10.1007/s10439-025-03701-6 40025252 PMC12123582 · DOI
  10. Succi MD, Chang BS, Rao AS. Building the AI-enabled medical school of the future. JAMA. 2025; 333(19): 1665- 1666. doi: 10.1001/jama.2025.2789 40163081 · DOI
  11. Rao AS, Kim J, Mu A,. Synthetic medical education in dermatology leveraging generative artificial intelligence. NPJ Digit Med. 2025; 8(1): 247. doi: 10.1038/s41746-025-01650-x 40320492 PMC12050279 · DOI
  12. Rao AS, Prasad S, Lee RS, Farrell S, McKinley S, Succi MD. Development and evaluation of an artificial intelligence-powered surgical oral examination simulator: a pilot study. Mayo Clin Proc Digit Health. 2025; 3(3): 100241. doi: 10.1016/j.mcpdig.2025.100241 40677929 PMC12270061 · DOI
  13. Rao AS, Lee RS, Bott E,. The digital standardized patient: an artificial intelligence coach for cultural dexterity in surgical care. J Am Coll Surg. 2025; 241(5): 887- 894. doi: 10.1097/XCS.0000000000001505 40704693 PMC13198390 · DOI
  14. Shieh A, Tran B, He G, Kumar M, Freed JA, Majety P. Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports. Sci Rep. 2024; 14(1): 9330. doi: 10.1038/s41598-024-58760-x 38654011 PMC11039662 · DOI · Paper page
  15. Young CC, Enichen E, Rao A,. Pilot study of large language models as an age-appropriate explanatory tool for chronic pediatric conditions. medRxiv. Preprint posted online August 7, 2024. doi: 10.1101/2024.08.06.24311544 · DOI

Cited by

Metadata

Authors
Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D.
Affiliations
  • Harvard Medical School, Boston, Massachusetts
  • Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center, Mass General Brigham, Boston, Massachusetts
Venue
JAMA Network Open
Published
2026-04-13
Volume
9
Issue
4
Pages
e264003
ISSN
2574-3805
Cited by
69
References
12
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.