CoChat

Large Language Model Performance and Clinical Reasoning Tasks

Rao, Arya S., Esmail, Kaiz P., Lee, Richard S., Jiang, Sharon, Arraiza Carlo, Bianca, Gill, Jasleen, Khanna, Praneet, Kalmowitz, Ezra, Montagnese, Basile, Heydari, Kimia, Jiao, Qiao, Bott, Ethan, Nguyen, Dan, Wang, Grace, Hood, Michael, Landman, Adam B., Succi, Marc D. · 2026 · JAMA Network Open
Cited by 69

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and Participants: In this cross-sectional study, performance was evaluated using standardized clinical vignettes from the January 2025 update of MSD Manual vignettes. A total of 21 off-the-shelf LLMs, including recently released GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4, were evaluated. Models were assessed by medical student scorers in triplicate across sequential stages of the standard clinical workflow. Analyses were performed from January to December 2025. Main Outcomes and Measures: The primary outcome was the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, defined as the normalized polygonal area representing balanced accuracy across 5 domains of clinical reasoning as follows: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous clinical reasoning questions. Analyses including analyses of variance, t tests, and regression models were used to compare AI model performance and demographic associations. Results: LLMs were tested across 29 clinical vignettes (representing 16 254 responses in total). PrIME-LLM scores ranged from 0.64 (range, 0.63-0.65) (Gemini 1.5 Flash) to 0.78 (range, 0.77-0.79) (Grok 4), with reasoning-optimized models outperforming nonreasoning models and GPT models scoring highest overall. Differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate. Failure rates exceeded 0.80 (range, 0.90-1.00) for differential diagnosis in all models but were less than 0.40 (range, 0.09-0.39) for final diagnosis. Multimodal performance was robust; most LLM models showed improved accuracy with image inputs. Conclusions and Relevance: In this cross-sectional study of 21 LLMs, frontier LLMs achieved high accuracy on final diagnoses but performed poorly in generating differential diagnoses and navigating uncertainty relative to other reasoning stages. The PrIME-LLM framework provided greater separation than raw accuracy, revealing critical reasoning gaps obscured by traditional benchmarks. Thus, despite version-based improvements and advantages in reasoning-optimized models, off-the-shelf LLMs have not yet achieved the intelligence required for safe deployment and remain limited in demonstrating advanced clinical reasoning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2026.4003

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores and Comparison of PrIME-LLM Scores and Overall Accuracy for Tested Clinical Vignettes A, Radar plots show accuracy across 5 domains of clinical reasoning: differential diagnosis (DD), diagnostic testing (DT), final diagnosis (FD), management (M), and miscellaneous clinical reasoning questions (Q). PrIME-LLM scores were calculated as the normalized polygonal area, summarizing balanced performance across tasks. B, Bar graphs show a comparison of overall accuracy vs PrIME-LLM scores for the tested clinical vignettes. Error bars indicate SEMs.
Figure 1. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores and Comparison of PrIME-LLM Scores and Overall Accuracy for Tested Clinical Vignettes A, Radar plots show accuracy across 5 domains of clinical reasoning: differential diagnosis (DD), diagnostic testing (DT), final diagnosis (FD), management (M), and miscellaneous clinical reasoning questions (Q). PrIME-LLM scores were calculated as the normalized polygonal area, summarizing balanced performance across tasks. B, Bar graphs show a comparison of overall accuracy vs PrIME-LLM scores for the tested clinical vignettes. Error bars indicate SEMs.CC BY · PubMed Central
Figure 2. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores by Model Family and PrIME-LLM Scores by Release Date A, Bar graph shows mean PrIME-LLM scores grouped by developer family (GPT, Claude, DeepSeek, Gemini, and Grok). Error bars indicate SEMs. B, Scatter plot shows version-based trajectories within each model family, showing successive releases in chronological order. ANOVA indicates analysis of variance.
Figure 2. Proportional Index of Medical Evaluation for Large Language Model (PrIME-LLM) Scores by Model Family and PrIME-LLM Scores by Release Date A, Bar graph shows mean PrIME-LLM scores grouped by developer family (GPT, Claude, DeepSeek, Gemini, and Grok). Error bars indicate SEMs. B, Scatter plot shows version-based trajectories within each model family, showing successive releases in chronological order. ANOVA indicates analysis of variance.CC BY · PubMed Central
Figure 3. Bar Graphs of Failure Rates of Tested Large Language Models A, Failure rate by age group, defined as the proportion of questions not answered fully correctly. B, Failure rate by patient sex. DD indicates differential diagnosis; DT, diagnostic testing; FD, final diagnosis; M, management; Q, miscellaneous clinical reasoning questions.
Figure 3. Bar Graphs of Failure Rates of Tested Large Language Models A, Failure rate by age group, defined as the proportion of questions not answered fully correctly. B, Failure rate by patient sex. DD indicates differential diagnosis; DT, diagnostic testing; FD, final diagnosis; M, management; Q, miscellaneous clinical reasoning questions.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 15

  1. Koirala, 2025, Evaluating AI performance in nephrology triage and subspecialty referrals., Sci Rep, 15, 3455, 10.1038/s41598-025-88074-5 · DOI
  2. Ong, 2024, SurgeryLLM: a retrieval-augmented generation large language model framework for surgical decision support and workflow enhancement., NPJ Digit Med, 7, 364, 10.1038/s41746-024-01391-3 · DOI
  3. Koranteng, 2023, Empathy and equity: key considerations for large language model adoption in health care., JMIR Med Educ, 9, 10.2196/51199 · DOI
  4. Rao, 2024, Proactive polypharmacy management using large language models: opportunities to enhance geriatric care., J Med Syst, 48, 41, 10.1007/s10916-024-02058-y · DOI · Paper page
  5. Young, 2025, Diagnostic accuracy of a custom large language model on rare pediatric disease case reports., Am J Med Genet A, 197, 10.1002/ajmg.a.63878 · DOI
  6. Young, 2025, Racial, ethnic, and sex bias in large language model opioid recommendations for pain management., Pain, 166, 511, 10.1097/j.pain.0000000000003388 · DOI
  7. Nguyen, 2025, Exploring the accuracy of embedded ChatGPT-4 and ChatGPT-4o in generating BI-RADS scores: a pilot study in radiologic clinical support., Clin Imaging, 117, 10.1016/j.clinimag.2024.110335 · DOI
  8. Kim, 2025, Risk stratification of potential drug interactions involving common over-the-counter medications and herbal supplements by a large language model., J Am Pharm Assoc (2003), 65, 10.1016/j.japh.2024.102304 · DOI
  9. Rao, 2025, A future of self-directed patient internet research: large language model-based tools versus standard search engines., Ann Biomed Eng, 53, 1199, 10.1007/s10439-025-03701-6 · DOI
  10. Succi, 2025, Building the AI-enabled medical school of the future., JAMA, 333, 1665, 10.1001/jama.2025.2789 · DOI
  11. Rao, 2025, Synthetic medical education in dermatology leveraging generative artificial intelligence., NPJ Digit Med, 8, 247, 10.1038/s41746-025-01650-x · DOI
  12. Rao, 2025, Development and evaluation of an artificial intelligence-powered surgical oral examination simulator: a pilot study., Mayo Clin Proc Digit Health, 3, 10.1016/j.mcpdig.2025.100241 · DOI
  13. Rao, 2025, The digital standardized patient: an artificial intelligence coach for cultural dexterity in surgical care., J Am Coll Surg, 241, 887, 10.1097/XCS.0000000000001505 · DOI
  14. Shieh, 2024, Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports., Sci Rep, 14, 9330, 10.1038/s41598-024-58760-x · DOI · Paper page
  15. Young? CC, Enichen? E, Rao? A, . Pilot study of large language models as an age-appropriate explanatory tool for chronic pediatric conditions.? medRxiv. Preprint posted online August 7, 2024. doi:10.1101/2024.08.06.24311544 · DOI

Cited by

See all 69

Metadata

Authors
Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D.
Affiliations
  • Harvard Medical School, Boston, Massachusetts
  • Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center, Mass General Brigham, Boston, Massachusetts
Venue
JAMA Network Open
Published
2026-04-13
Volume
9
Issue
4
Pages
e264003
ISSN
2574-3805
Cited by
69
References
12
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large Language Model Performance and Clinical Reasoning Tasks. https://app.cochat.ai/papers/10.1001/jamanetworkopen.2026.4003
The paper
Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D. (2026). Large Language Model Performance and Clinical Reasoning Tasks. JAMA Network Open. https://doi.org/10.1001/jamanetworkopen.2026.4003

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.