An evaluation framework for clinical use of large language models in patient interaction tasks
Abstract
The integration of large language models (LLMs) into clinical diagnostics has the potential to transform doctor-patient interactions. However, the readiness of these models for real-world clinical application remains inadequately tested. This paper introduces the Conversational Reasoning Assessment Framework for Testing in Medicine (CRAFT-MD) approach for evaluating clinical LLMs. Unlike traditional methods that rely on structured medical examinations, CRAFT-MD focuses on natural dialogues, using simulated artificial intelligence agents to interact with LLMs in a controlled environment. We applied CRAFT-MD to assess the diagnostic capabilities of GPT-4, GPT-3.5, Mistral and LLaMA-2-7b across 12 medical specialties. Our experiments revealed critical insights into the limitations of current LLMs in terms of clinical conversational reasoning, history-taking and diagnostic accuracy. These limitations also persisted when analyzing multimodal conversational and visual assessment capabilities of GPT-4V. We propose a comprehensive set of recommendations for future evaluations of clinical LLMs based on our empirical findings. These recommendations emphasize realistic doctor-patient conversations, comprehensive history-taking, open-ended questioning and using a combination of automated and expert evaluations. The introduction of CRAFT-MD marks an advancement in testing of clinical LLMs, aiming to ensure that these models augment medical practice effectively and ethically.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous benchmarks like AMIE, SAPS, and CRAFT-MD focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.
“Previous benchmarks like AMIE23, SAPS30, and CRAFT-MD31 focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [31]
STARD-AI offers a strong foundation for transparent reporting, but complementary frameworks such as CRAFT-MD may be better suited for evaluating different forms of AI-driven clinical support.
“Although STARD-AI offers a strong foundation for transparent reporting, complementary frameworks such as CRAFT-MD may be better suited for evaluating different forms of AI-driven clinical support 53 .”
From The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence (2025) · cites this paper as [53]
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212 · see citing works
Self-citationAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
Self-citationBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Irving, G., Neves, A. L., Dambha-Miller, H., Oishi, A., Tagashira, H., Verho, A., & Holden, J. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017902 · Cited by 1,083 · see citing works
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)Preprint
Yang, Z.-Y., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2309.17421 · Cited by 1,036 · see citing works
The State of Telehealth Before and After the COVID-19 Pandemic
Shaver, J. (2022) · Primary Care: Clinics in Office Practice · doi:10.1016/j.pop.2022.04.002 · Cited by 863 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Lasser, K. E., Himmelstein, D. U., & Woolhandler, S. J. (2006) · American Journal of Public Health · doi:10.2105/ajph.2004.059402 · Cited by 604 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Role play with large language models
Shanahan, M., McDonell, K., & Reynolds, L. (2023) · Nature · doi:10.1038/s41586-023-06647-8 · Cited by 491 · see citing works
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443 · see citing works
Self-citationCreation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J., Shin, J. H., Sullivan, P. L. Z., Cielo, D., Oyelese, A. A., Doberstein, C. E., Telfeian, A. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002551 · Cited by 354 · see citing works
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Dermatology in primary care: prevalence and patient disposition.
Lowell, B. A., Froelich, C. W., Federman, D. G., & Kirsner, R. S. (2001) · Journal of the American Academy of Dermatology · doi:10.1067/mjd.2001.114598 · Cited by 231 · see citing works
Can ChatGPT pass the life support exams without entering the American heart association course?
Fijačko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109732 · Cited by 206 · see citing works
AI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
Chatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.3386/w31422 · Cited by 161 · see citing works
Self-citationCombining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161 · see citing works
Self-citationMoshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124 · see citing works
Lin, J. C. H., Younessi, D. N., Kurapati, S. S., Tang, O. Y., & Scott, I. U. (2023) · Eye · doi:10.1038/s41433-023-02564-2 · Cited by 118 · see citing works
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · see citing works
Towards Conversational Diagnostic AIPreprint
Tu, T., Palepu, A. K., Schaekermann, M., Saab, K. K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomašev, N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.05654 · Cited by 114 · see citing works
Giannos, P. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000451 · Cited by 84 · see citing works
Ali, R., Tang, O. Y., Connolly, I. D., et al. (2023) · medRxiv · doi:10.1101/2023.04.06.23288265 · Cited by 79 · see citing works
DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving AgentsPreprint
Nair, V., Schumacher, E., Tso, G., et al. (2023) · Clinical Natural Language Processing Workshop · doi:10.48550/arxiv.2303.17071 · Cited by 75 · see citing works
Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment
Lai, U. H., Wu, K. S., Hsu, T.-Y., & Kan, J. K. C. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1240915 · Cited by 67 · see citing works
Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2023) · doi:10.1101/2023.05.04.23289493 · Cited by 45 · see citing works
Evaluating GPT-4-based ChatGPT's clinical potential on the NEJM quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2024) · BMC Digital Health · doi:10.1186/s44247-023-00058-5 · Cited by 45 · see citing works
de Zarzà, I., de Curtò, J., Roig, G., Manzoni, P., & Calafate, C. T. (2023) · Electronics · doi:10.3390/electronics12122722 · Cited by 41 · see citing works
DERA: Enhancing large language model completions with dialog-enabled resolving agents
Nair, V., Schumacher, E., Tso, G., & Kannan, A. (2024) · Proceedings of the 6th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2024.clinicalnlp-1.12 · Cited by 38 · see citing works
Performance of ChatGPT on Chinese master’s degree entrance examination in clinical medicine
Li, K.-C., Bu, Z.-J., Shahjalal, M., He, B.-X., Zhuang, Z.-F., Li, C., Liu, J.-P., Wang, B., & Liu, Z.-L. (2024) · PLOS ONE · doi:10.1371/journal.pone.0301702 · Cited by 23 · see citing works
Multimodal Foundation Models Exploit Text to Make Medical Image PredictionsPreprint
Buckley, T., Diao, J. A., Rajpurkar, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05591 · Cited by 18 · see citing works
Self-citationMultimodal foundation models exploit text to make medical image predictions
Buckley, T. A., Diao, J. A., Srivastava, C. N., Brodeur, P. G., Rajpurkar, P., Rodman, A. M., & Manrai, A. K. (2026) · Nature Communications · doi:10.1038/s41467-026-74207-5 · Cited by 18 · see citing works
Self-citationDermatology consultations: how long do they take?
Wong, J. L. C., Vincent, R. C., & Al-Sharqi, A. (2017) · Future Hospital Journal · doi:10.7861/futurehosp.4-1-23 · Cited by 13 · see citing works
Goldberger, T., & Armoni-Weiss, G. (2023) · New England Journal of Medicine · doi:10.1056/nejmicm2210388 · Cited by 3 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Cited by
Sorted by citations
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366 · see citing works
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence
Sounderajah, V., Guni, A., Liu, X., Collins, G. S., Karthikesalingam, A., Markar, S. R., Golub, R. M., Denniston, A. K., Shetty, S., Moher, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03953-8 · Cited by 352 · see citing works
Performance of a large language model on the reasoning tasks of a physician
Brodeur, P. G., Buckley, T. A., Kanjee, Z., Goh, E., Ling, E., Jain, P., Cabral, S., Abdulnour, R.-E. E., Haimovich, A. D., Freed, J. A., et al. (2026) · Science · doi:10.1126/science.adz4433 · Cited by 135 · see citing works
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara Hincapié, M., Ekanayaka, A. S., Tarassenko, L., Rocher, L., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04074-y · Cited by 130 · see citing works
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 118 · see citing works
Self-citationEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 87 · see citing works
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84 · see citing works
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 76 · see citing works
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 71 · see citing works
Barman, K. G., Wood, N. G., & Pawłowski, P. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09778-2 · Cited by 69 · see citing works
Assessment of Large Language Models in Clinical Reasoning: A Novel Benchmarking Study
McCoy, L. G., Swamy, R., Sagar, N., Wang, M., Bacchi, S., Fong, J. M. N., Tan, N. C., Tan, K., Buckley, T. A., Brodeur, P. G., et al. (2025) · NEJM AI · doi:10.1056/aidbp2500120 · Cited by 69 · see citing works
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 66 · see citing works
Coordinated AI agents for advancing healthcare
Moritz, M., Topol, E. J., & Rajpurkar, P. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01363-2 · Cited by 59 · see citing works
Self-citationThe evaluation illusion of large language models in medicine
Agrawal, M., Chen, I. Y., Gulamali, F., & Joshi, S. D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01963-x · Cited by 59 · see citing works
Zhou, J., Zhang, J., Wan, R., Cui, X., Liu, Q., Guo, H., Shi, X., Fu, B., Meng, J., Yue, B., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06916-2 · Cited by 55 · see citing works
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53 · see citing works
Red teaming ChatGPT in medicine to yield real-world insights on model behavior
Chang, C. T., Farah, H., Gui, H., Rezaei, S. J., Bou-Khalil, C., Park, Y., Swaminathan, A., Omiye, J. A., Kolluri, A., Chaurasia, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01542-0 · Cited by 51 · see citing works
Self-citationA survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 50 · see citing works
Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 49 · see citing works
Assessing and alleviating state anxiety in large language models
Ben‐Zion, Z., Witte, K., Jagadish, A. K., Duek, O. A., Harpaz‐Rotem, I., Khorsandian, M.-C., Burrer, A., Seifritz, E., Homan, P., Schulz, E., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01512-6 · Cited by 46 · see citing works
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 45 · see citing works
Park, T., Lee, I., Lee, S. W., & Kong, S. W. (2025) · Korean Journal of Pediatrics · doi:10.3345/cep.2025.00962 · Cited by 39 · see citing works
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 35 · see citing works
Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02277-8 · Cited by 33 · see citing works
Benchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 31 · see citing works
Giacobbe, D. R., Marelli, C., Manna, B. L., Padua, D., Malva, A., Guastavino, S., Signori, A., Mora, S., Rosso, N., Campi, C., et al. (2025) · npj Antimicrobials and Resistance · doi:10.1038/s44259-025-00084-5 · Cited by 29 · see citing works
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Baur, D., Ansorg, J., Heyde, C., & Voelker, A. (2025) · JMIR AI · doi:10.2196/75262 · Cited by 26 · see citing works
Beyond Assistance: The Case for Role Separation in AI-Human Radiology Workflows
Rajpurkar, P., & Topol, E. J. (2025) · Radiology · doi:10.1148/radiol.250477 · Cited by 26 · see citing works
Self-citationGómez, C., Yin, J., Huang, C., & Unberath, M. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1427463 · Cited by 23 · see citing works
Livingston, L., Featherstone-Uwague, A., Barry, A., Barretto, K., Morey, T., Herrmannová, D., & Avula, V. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf054 · Cited by 24 · see citing works
Streamlining evidence based clinical recommendations with large language models
Li, D., Jiang, N., Huang, K., Tu, R., Ouyang, S., Yu, H., Lin, Q., Chen, Y., Zhou, T., Tong, D., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02273-y · Cited by 20 · see citing works
Large language models for frontline healthcare support in low-resource settings
Rutunda, S., Williams, G., Kabanda, K., Nkurunziza, F., Uwiduhaye, S., Rugegamanzi, E., Nshimiyimana, C., Menon, V., Emmanuel-Fabula, M., Denniston, A. K., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00038-1 · Cited by 20 · see citing works
Large Language and Vision Assistant in dermatology: a game changer or just hype?
Göktaş, P., Gülseren, D., & Tobin, A. (2024) · Clinical and Experimental Dermatology · doi:10.1093/ced/llae119 · Cited by 20 · see citing works
Haider, S. A., Prabha, S., Gomez-Cabello, C. A., Borna, S., Genovese, A., Trabilsy, M., Collaço, B. G., Wood, N. G., Bagaria, S. P., Tao, C., et al. (2025) · Sensors · doi:10.3390/s25144305 · Cited by 19 · see citing works
When It Comes to Benchmarks, Humans Are the Only Way
Rodman, A. M., Zwaan, L., Olson, A. P. J., & Manrai, A. K. (2025) · NEJM AI · doi:10.1056/aie2500143 · Cited by 19 · see citing works
Artificial intelligence in primary care: innovation at a crossroads
Laranjo, L., Car, L. T., Payne, R. E., Neves, A. L., Kidd, M. R., & Miranda, J. J. (2025) · The Lancet Primary Care · doi:10.1016/j.lanprc.2025.100078 · Cited by 18 · see citing works
Principles to guide clinical AI readiness and move from benchmarks to real-world evaluation
Azad, T. D., Krumholz, H. M., & Saria, S. (2026) · Nature Medicine · doi:10.1038/s41591-025-04198-1 · Cited by 17 · see citing works
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A. M., Cai, T., Dagan, N., Balicer, R. D., Loscalzo, J., Kohane, I. S., & Žitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 17 · see citing works
Hashjin, N. M., Amiri, M. H., & Najafabadi, M. K. (2026) · Scientific Reports · doi:10.1038/s41598-026-38715-0 · Cited by 17 · see citing works
Ohde, J. W., Thompson, A., Liu, Z., Rost, L. M., Overgaard, J. D., Tan, J. Y. E., Overgaard, S. M., Sarmiento, R. F., Ke, Y., Liew, J. C. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02554-0 · Cited by 17 · see citing works
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 14 · see citing works
Understanding the robustness of vision-language models to medical image artefacts
Cheng, Z., Ong, A. Y., Wagner, S. K., Merle, D. A., Ju, L., Zhang, H., Chen, R., Pang, L., Li, B., He, T., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02108-w · Cited by 14 · see citing works
Robustness tests for biomedical foundation models should tailor to specifications
Xian, R. P., Baker, N., David, T., Cui, Q., Holmgren, A. J., Bauer, S., Sushil, M., & Abbasi-Asl, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01926-2 · Cited by 14 · see citing works
Ten Natural Language Processing Tasks with Generative Artificial Intelligence
Golec, J., & Hachaj, T. (2025) · Applied Sciences · doi:10.3390/app15169057 · Cited by 13 · see citing works
Legal Dimensions of AI Contracts in Sports Talent Management: Challenges and SolutionsPreprint
Asiabar, M. G., Asiabar, M. G., & Asiabar, A. G. (2025) · doi:10.14293/pr2199.001393.v1 · Cited by 13 · see citing works
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.266 · Cited by 13 · see citing works
Ghorbian, M., Ghobaei-Arani, M., & Ghorbian, S. (2025) · Methods · doi:10.1016/j.ymeth.2025.04.001 · Cited by 14 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
Chen, M., Wu, Y., Ma, J., Jia, X., Gao, C., Zhao, F., & Qiao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02409-8 · Cited by 13 · see citing works
Allison, D. M., & Herzog, S. (2025) · Risk Analysis · doi:10.1111/risa.70105 · Cited by 9 · see citing works
Si, Y., Meng, Y., Chen, X., An, R., Mao, L., Li, B., Bateman, H., Zhang, H., Fan, H., Zu, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01956-w · Cited by 9 · see citing works
AI and Digital Tools in Dermatology: Addressing Access and Misinformation
du Crest, D., Madhumita, M., Enbiale, W., Postigo, J. A. R., Malvehy, J., Wongvibulsin, S., Gupta, S., Kittler, H., Skayem, C., Mahto, A., et al. (2026) · JMIR Dermatology · doi:10.2196/79044 · Cited by 9 · see citing works
Self-citationImran, M., & Lee, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3656368 · Cited by 10 · see citing works
A clinical environment simulator for dynamic AI evaluation
Luo, L., Kim, S. E., Zhang, X., Kernbach, J. M., Kenia, R., Acosta, J. N., Nathanson, L. A., Haimovich, A. D., Rodman, A. M., Goh, E., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04252-6 · Cited by 11 · see citing works
Self-citationSirago, G., Calò, F., Vinci, A., Visci, P., Solarino, B., Dell'Erba, A. S., & Ferorelli, D. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02363-8 · Cited by 9 · see citing works
Zero-Shot Large Language Models for Long Clinical Text Summarization with Temporal ReasoningPreprint
Kruse, M., Hu, S., Derby, N., Wu, Y., Stonbraker, S., Yao, B., Wang, D., Goldberg, E., & Gao, Y. (2025) · medRxiv · doi:10.1101/2025.07.21.25331947 · Cited by 9 · see citing works
A Patient-in-the-Loop Approach to Artificial Intelligence in Medicine
Griot, M., & Walker, G. A. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.14460 · Cited by 8 · see citing works
The Financial, Operational, and Clinical Advantages of Generalist Radiology AI
Dogra, S., Zhang, X., Silva, E., & Rajpurkar, P. (2025) · Radiology · doi:10.1148/radiol.242362 · Cited by 8 · see citing works
Self-citationWill Autonomous AI Exceed AI-Aided Physicians as the Best Medical Care?
Emanuel, E., Baker-Butler, A., Khosla, N., & Khosla, V. (2026) · JAMA · doi:10.1001/jama.2026.15380 · Cited by 8 · see citing works
Huh, C. Y., Lee, J., Kim, G., Jang, Y., Ko, H.-S., Suh, M. J., Hwang, S., Son, H. J., Song, J., Kim, S., et al. (2025) · Information · doi:10.3390/info16080653 · Cited by 7 · see citing works
Foundation models in clinical oncology: Progresses and perspectives
Li, J., Fu, J., Geng, X., & Wang, H. (2025) · Cancer Letters · doi:10.1016/j.canlet.2025.218220 · Cited by 7 · see citing works
Misra, R., Dahal, R., Kirk, B., Khan, R., Dogan, G., Chataut, R., & Gyawali, P. (2026) · International Journal of Qualitative Methods · doi:10.1177/16094069261426100 · Cited by 7 · see citing works
Artificial intelligence–enabled precision medicine for inflammatory skin diseases
Tang, A. S., Wei, M. L., Haemel, A., La, C., Sirota, M., & Lee, E. Y. (2026) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2025.10.596 · Cited by 6 · see citing works
ChatSOS: Vector database augmented generative question answering assistant in safety engineering
Tang, H., Chen, D., Chu, Q., & Liu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128644 · Cited by 6 · see citing works
Zhu, X., Dai, W., Evans, R. D., Geng, X., Mu, A., & Liu, Z. (2025) · JMIR Medical Informatics · doi:10.2196/76636 · Cited by 6 · see citing works
A bibliometric analysis of artificial intelligence in medical education (2015–2025)
Cheng, W., Hu, Z., & Yu, H. (2025) · Medicine · doi:10.1097/md.0000000000045684 · Cited by 6 · see citing works
Artificial intelligence in medicine: a position paper by the Italian Society of Internal Medicine
Balsano, C., Cabitza, F., Cicco, S., Gori, M., Malerba, D., Montagna, M., Tarquini, R., Vacca, A., & The Working Group on Artificial Intelligence, D. T. O. T. I. S. O. I. M. (. (2025) · Internal and Emergency Medicine · doi:10.1007/s11739-025-04146-4 · Cited by 6 · see citing works
Debsarkar, S. S., & Prasath, V. B. S. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102704 · Cited by 6 · see citing works
Gan, Y., Guo, J., Zhai, J., & Huang, J. (2026) · Advances in Medical Education and Practice · doi:10.2147/amep.s573041 · Cited by 6 · see citing works
Patients Facing Large Language Models in Oncology: A Narrative Review
Raynaud, C., Wu, D., Levy, J., Marengo, M., & Bibault, J. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00149 · Cited by 6 · see citing works
MAP: evaluation and multi-agent enhancement of large language models for inpatient pathways
Chen, Z., Peng, Z., Liang, X., Wang, C., Liang, P., Zeng, L., Ju, M., & Yuan, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00085-0 · Cited by 6 · see citing works
Duan, Z., Huang, X., Lu, R., Xu, W., Liu, H., Geng, Y., Takahashi, N., Wu, Y., Wang, Q., Song, Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102465 · Cited by 5 · see citing works
Rethinking AI Workflows: Guidelines for Scientific Evaluation in Digital Health Companies
McAlister, K. L., Gonzales, L., & Huberty, J. (2025) · JMIR AI · doi:10.2196/71798 · Cited by 5 · see citing works
Evaluating the Insights of ChatGPT, Gemini and Expert Surgeons in Revision Rhinoplasty Consultation
Bastaninejad, S., Alipour, S., Ishida, L. C., Mohebbi, M., Mousavi-Asl, B., Heidari, F., & Azimi, H. (2025) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05464-w · Cited by 5 · see citing works
Weimann, T. G., Schlieter, H., Konofalska, U., & Ziemssen, T. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2641598 · Cited by 5 · see citing works
Wang, T.-H., Jheng, J.-C., Tseng, Y.-T., Chen, L., & Chen, Y. (2025) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000001273 · Cited by 5 · see citing works
Performance of o1 pro and GPT-4 in Self-Assessment Questions for Nephrology Board Renewal
Noda, R., Yuasa, C., Kitano, F., Ichikawa, D., & Shibagaki, Y. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1702668 · Cited by 5 · see citing works
Artificial intelligence-enabled precision medicine for inflammatory skin diseasesPreprint
Tang, A., Wei, M. L., Haemel, A. K., La, C., Sirota, M., & Lee, E. Y. (2025) · PubMed · doi:10.48550/arxiv.2505.09527 · Cited by 5 · see citing works
Generative AI in healthcare: challenges to patient agency and ethical implications
Holmes, S. A., Faria, V., & Moulton, E. A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1524553 · Cited by 4 · see citing works
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4 · see citing works
Liang, J., Xing, M., Xiang, P., Wang, G., Chen, M., Fang, Q., Zhou, T., Lu, Z., Leng, X., Huang, J., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02537-1 · Cited by 4 · see citing works
Automating evaluation of LLM-generated responses to patient questions about rare diseases
Zhao, M., Oh, I. Y., Gupta, A., Cohen‐Cutler, S., Harmoney, K. M., Lai, A. M., & Sisk, B. A. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag054 · Cited by 4 · see citing works
Cronin, A., Kelly, A., Wrona, M., O'Donnell, P., Hassan, A. A., Myles, T., Fallon, T., & MacFarlane, A. E. (2025) · BJGP Open · doi:10.3399/bjgpo.2025.0107 · Cited by 5 · see citing works
Shi, Z., Wu, B., Hu, B., Zhong, J., Li, Z., Zhang, F., Chen, Z., Yang, C., Guo, B., Xu, Q., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66910-6 · Cited by 4 · see citing works
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 4 · see citing works
A S.C.O.R.E. framework for evaluating open-ended responses from large language models in healthcare
Tan, T. F., Elangovan, K., ONG, J., Ke, Y., Lee, A., Shah, N. H., Sung, J., Wong, T. Y., Xue, L., Liu, N., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102883 · Cited by 4 · see citing works
Mahajan, A., & Lees, A. J. (2025) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.70054 · Cited by 4 · see citing works
The application and challenges of ChatGPT in laboratory medicine
Niu, Z., Kuang, X., Chen, J., Cai, X., & Zhang, P. (2025) · Advances in Laboratory Medicine / Avances en Medicina de Laboratorio · doi:10.1515/almed-2025-0080 · Cited by 3 · see citing works
Zhu, S., Xie, Y., Tang, Y., Zhou, Y., Zhao, R., & Dong, X. (2025) · European Journal of Pediatrics · doi:10.1007/s00431-025-06602-x · Cited by 3 · see citing works
Applications of artificial intelligence and large language models in cancer immunotherapy
Su, Y., Liu, J., Li, J., Huang, X., Hajdú, A., Fu, R., Xu, B., Wang, Y., Liu, X., Li, Z., et al. (2025) · Current Proteomics · doi:10.1016/j.curpro.2025.100057 · Cited by 3 · see citing works
Grounding large language models in clinical diagnostics
Chen, X., Zhou, H., Yi, H., You, M., Liu, W., Li, W., Qin, Z., Li, H., Zhang, X., Guo, Y., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70274-w · Cited by 3 · see citing works
Self-citationByun, H., Lee, D., Jung, M., & Jang, B. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02440-y · Cited by 3 · see citing works
A Platform for Interactive AI Character Experiences
Wampfler, R., Yang, C., Elste, D., Kovačević, N., Witzig, P., & Groß, M. H. (2025) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3721238.3730762 · Cited by 3 · see citing works
Evaluating Large Reasoning Model Performance on Complex Medical Scenarios In The MMLU-Pro BenchmarkPreprint
Hoyt, R. E., Knight, D. R. T., Haider, M., & Bajwa, M. (2025) · medRxiv · doi:10.1101/2025.04.07.25325385 · Cited by 3 · see citing works
Steinbuch, S., de Vos‐Hillebrand, L., O'Neill-Dee, C., Wu, I., Steinbuch, H., Kulcsár, Z., Ranjan, A., Verma, A., Landsberg, J., Dietrich, D., et al. (2026) · medRxiv · doi:10.64898/2026.01.23.26344723 · Cited by 3 · see citing works
Auditing medical multi-agent AI reveals risks of false consensus
Zhu, Y., Gu, L., Wang, Z., Sang, H., Sui, D., Tang, W., Mi, L., Wang, Y., Gao, J., Yao, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.10185 · Cited by 2 · see citing works
Souza, G. D. P., Melo, G., & Schneider, D. (2025) · Informatics · doi:10.3390/informatics12040113 · Cited by 2 · see citing works
Wen, Z., Huang, J., Yu, K., Li, Y., Wang, Z., Liao, X., Li, B., Tao, Z., & He, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-30050-0 · Cited by 2 · see citing works
- Authors
- Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., Daneshjou, R., & Rajpurkar, P.
- Affiliations
- Harvard University (US)
- Washington Hospital (US)
- Northwestern University (US)
- University of California, Los Angeles (US)
- Stanford University (US)
- Stanford Medicine (US)
- Corresponding author
- Daneshjou, Roxana
- Rajpurkar, Pranav
- ORCID
- Venue
- Nature Medicine
- Published
- 2025-01
- Volume
- 31
- Issue
- 1
- Pages
- 77-86
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4405995927
- Semantic Scholar
- 597f3cc7c41ba1da58624f79771b26a5fd8de1ba
- Cited by
- 201
- References
- 45
- Citations by year
- 2026: 114 · 2025: 86 · 2024: 1
- FWCI
- 47.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, clinical evaluation, doctor-patient interaction, medical history, diagnostic accuracy, GPT-4, GPT-3.5, Mistral, Llama 2, GPT-4V, multimodal assessment, evaluation framework
- MeSH terms
- Large Language Models, Artificial Intelligence, Communication, Humans, Medical History Taking, Physician-Patient Relations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
