Large Language Model Performance and Clinical Reasoning Tasks
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and Participants: In this cross-sectional study, performance was evaluated using standardized clinical vignettes from the January 2025 update of MSD Manual vignettes. A total of 21 off-the-shelf LLMs, including recently released GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4, were evaluated. Models were assessed by medical student scorers in triplicate across sequential stages of the standard clinical workflow. Analyses were performed from January to December 2025. Main Outcomes and Measures: The primary outcome was the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, defined as the normalized polygonal area representing balanced accuracy across 5 domains of clinical reasoning as follows: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous clinical reasoning questions. Analyses including analyses of variance, t tests, and regression models were used to compare AI model performance and demographic associations. Results: LLMs were tested across 29 clinical vignettes (representing 16 254 responses in total). PrIME-LLM scores ranged from 0.64 (range, 0.63-0.65) (Gemini 1.5 Flash) to 0.78 (range, 0.77-0.79) (Grok 4), with reasoning-optimized models outperforming nonreasoning models and GPT models scoring highest overall. Differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate. Failure rates exceeded 0.80 (range, 0.90-1.00) for differential diagnosis in all models but were less than 0.40 (range, 0.09-0.39) for final diagnosis. Multimodal performance was robust; most LLM models showed improved accuracy with image inputs. Conclusions and Relevance: In this cross-sectional study of 21 LLMs, frontier LLMs achieved high accuracy on final diagnoses but performed poorly in generating differential diagnoses and navigating uncertainty relative to other reasoning stages. The PrIME-LLM framework provided greater separation than raw accuracy, revealing critical reasoning gaps obscured by traditional benchmarks. Thus, despite version-based improvements and advantages in reasoning-optimized models, off-the-shelf LLMs have not yet achieved the intelligence required for safe deployment and remain limited in demonstrating advanced clinical reasoning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a pattern mirroring our observation that topics with sparse or uncertain evidence produced the highest hallucination rates (26).”
From Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care (2026) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
Self-citationLeveraging large language models for decision support in personalized oncology
Benary, M., Wang, X. D., Schmidt, M., Soll, D., Hilfenhaus, G., Nassir, M., Sigler, C., Knödler, M., Keller, U., Beule, D., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.43689 · Cited by 306
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 172
The effect of using a large language model to respond to patient messages
Chen, S., Guevara, M., Moningi, S., Hoebers, F., Elhalawani, H., Kann, B. H., Chipidza, F. E., Leeman, J., Aerts, H. J. W. L., Miller, T., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00060-8 · Cited by 147
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135
Williams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119
Research works printed but not linked: 15
- Koirala, 2025, Evaluating AI performance in nephrology triage and subspecialty referrals., Sci Rep, 15, 3455, 10.1038/s41598-025-88074-5 · DOI
- Ong, 2024, SurgeryLLM: a retrieval-augmented generation large language model framework for surgical decision support and workflow enhancement., NPJ Digit Med, 7, 364, 10.1038/s41746-024-01391-3 · DOI
- Koranteng, 2023, Empathy and equity: key considerations for large language model adoption in health care., JMIR Med Educ, 9, 10.2196/51199 · DOI
- Rao, 2024, Proactive polypharmacy management using large language models: opportunities to enhance geriatric care., J Med Syst, 48, 41, 10.1007/s10916-024-02058-y · DOI · Paper page
- Young, 2025, Diagnostic accuracy of a custom large language model on rare pediatric disease case reports., Am J Med Genet A, 197, 10.1002/ajmg.a.63878 · DOI
- Young, 2025, Racial, ethnic, and sex bias in large language model opioid recommendations for pain management., Pain, 166, 511, 10.1097/j.pain.0000000000003388 · DOI
- Nguyen, 2025, Exploring the accuracy of embedded ChatGPT-4 and ChatGPT-4o in generating BI-RADS scores: a pilot study in radiologic clinical support., Clin Imaging, 117, 10.1016/j.clinimag.2024.110335 · DOI
- Kim, 2025, Risk stratification of potential drug interactions involving common over-the-counter medications and herbal supplements by a large language model., J Am Pharm Assoc (2003), 65, 10.1016/j.japh.2024.102304 · DOI
- Rao, 2025, A future of self-directed patient internet research: large language model-based tools versus standard search engines., Ann Biomed Eng, 53, 1199, 10.1007/s10439-025-03701-6 · DOI
- Succi, 2025, Building the AI-enabled medical school of the future., JAMA, 333, 1665, 10.1001/jama.2025.2789 · DOI
- Rao, 2025, Synthetic medical education in dermatology leveraging generative artificial intelligence., NPJ Digit Med, 8, 247, 10.1038/s41746-025-01650-x · DOI
- Rao, 2025, Development and evaluation of an artificial intelligence-powered surgical oral examination simulator: a pilot study., Mayo Clin Proc Digit Health, 3, 10.1016/j.mcpdig.2025.100241 · DOI
- Rao, 2025, The digital standardized patient: an artificial intelligence coach for cultural dexterity in surgical care., J Am Coll Surg, 241, 887, 10.1097/XCS.0000000000001505 · DOI
- Shieh, 2024, Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports., Sci Rep, 14, 9330, 10.1038/s41598-024-58760-x · DOI · Paper page
- Young? CC, Enichen? E, Rao? A, . Pilot study of large language models as an age-appropriate explanatory tool for chronic pediatric conditions.? medRxiv. Preprint posted online August 7, 2024. doi:10.1101/2024.08.06.24311544 · DOI
Cited by
Seifi, A., & Seyfi, A. (2026) · Critical Care Explorations · doi:10.1097/cce.0000000000001474 · Cited by 1
Yano, Y., Kakizaki, H., Nagasu, H., Kishi, S., Koshida, T., Nihei, Y., Hirano, A., Sugawara, Y., Imaizumi, T., Osakabe, Y., et al. (2026) · doi:10.64898/2026.07.15.26358121
Metadata
- Authors
- Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D.
- Affiliations
- Harvard Medical School, Boston, Massachusetts
- Medically Engineered Solutions in Healthcare Incubator, Innovation in Operations Research Center, Mass General Brigham, Boston, Massachusetts
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2026-04-13
- Volume
- 9
- Issue
- 4
- Pages
- e264003
- ISSN
- 2574-3805
- PMC
- PMC13077515
- Cited by
- 69
- References
- 12
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Model Performance and Clinical Reasoning Tasks. https://app.cochat.ai/papers/10.1001/jamanetworkopen.2026.4003
- The paper
- Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., Jiao, Q., Bott, E., Nguyen, D., Wang, G., Hood, M., Landman, A. B., & Succi, M. D. (2026). Large Language Model Performance and Clinical Reasoning Tasks. JAMA Network Open. https://doi.org/10.1001/jamanetworkopen.2026.4003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.