Cited by 224Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareMachine Learning in Healthcare

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.

An evaluation framework for clinical use of large language models in patient interaction tasks

Johri, Shreya, Jeong, Jaehwan, Tran, Benjamin A., Schlessinger, Daniel I., Wongvibulsin, Shannon, Barnes, Leandra A., Zhou, Hong-Yu, Cai, Zhuo Ran, Van Allen, Eliezer M., Kim, David, Daneshjou, Roxana, Rajpurkar, Pranav·2025·Nature Medicine

DOI 10.1038/s41591-024-03328-5

Open this paper in CoChat — free account

Abstract

The integration of large language models (LLMs) into clinical diagnostics has the potential to transform doctor-patient interactions. However, the readiness of these models for real-world clinical application remains inadequately tested. This paper introduces the Conversational Reasoning Assessment Framework for Testing in Medicine (CRAFT-MD) approach for evaluating clinical LLMs. Unlike traditional methods that rely on structured medical examinations, CRAFT-MD focuses on natural dialogues, using simulated artificial intelligence agents to interact with LLMs in a controlled environment. We applied CRAFT-MD to assess the diagnostic capabilities of GPT-4, GPT-3.5, Mistral and LLaMA-2-7b across 12 medical specialties. Our experiments revealed critical insights into the limitations of current LLMs in terms of clinical conversational reasoning, history-taking and diagnostic accuracy. These limitations also persisted when analyzing multimodal conversational and visual assessment capabilities of GPT-4V. We propose a comprehensive set of recommendations for future evaluations of clinical LLMs based on our empirical findings. These recommendations emphasize realistic doctor-patient conversations, comprehensive history-taking, open-ended questioning and using a combination of automated and expert evaluations. The introduction of CRAFT-MD marks an advancement in testing of clinical LLMs, aiming to ensure that these models augment medical practice effectively and ethically.

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03328-5

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
AI in health and medicine
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Self-citation
Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical ExamsPreprint
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Self-citation

Show all 45 references

Research works printed but not linked: 1

Image Challenge. N. Engl. J. Med. https://www.nejm.org/image-challenge (2024).

Sorted by citations

Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence
Sounderajah, V., Guni, A., Liu, X., Collins, G. S., Karthikesalingam, A., Markar, S. R., Golub, R. M., Denniston, A. K., Shetty, S., Moher, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03953-8 · Cited by 352
Performance of a large language model on the reasoning tasks of a physician
Brodeur, P. G., Buckley, T. A., Kanjee, Z., Goh, E., Ling, E., Jain, P., Cabral, S., Abdulnour, R.-E. E., Haimovich, A. D., Freed, J. A., et al. (2026) · Science · doi:10.1126/science.adz4433 · Cited by 187
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara Hincapié, M., Ekanayaka, A. S., Tarassenko, L., Rocher, L., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04074-y · Cited by 130
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 120
Self-citation
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 97
Large Language Models in Medical Chatbots: Opportunities, Challenges, and the Need to Address AI Risks
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 87
From large language models to multimodal AI: a scoping review on the potential of generative AI in medicine
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 79
Beyond transparency and explainability: on the need for adequate and contextualized user guidelines for LLM use
Barman, K. G., Wood, N. G., & Pawłowski, P. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09778-2 · Cited by 74
Assessment of Large Language Models in Clinical Reasoning: A Novel Benchmarking Study
McCoy, L. G., Swamy, R., Sagar, N., Wang, M., Bacchi, S., Fong, J. M. N., Tan, N. C., Tan, K., Buckley, T. A., Brodeur, P. G., et al. (2025) · NEJM AI · doi:10.1056/aidbp2500120 · Cited by 81
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 69
Coordinated AI agents for advancing healthcare
Moritz, M., Topol, E. J., & Rajpurkar, P. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01363-2 · Cited by 59
Self-citation
The evaluation illusion of large language models in medicine
Agrawal, M., Chen, I. Y., Gulamali, F., & Joshi, S. D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01963-x · Cited by 70
Integrating AI into clinical education: evaluating general practice trainees’ proficiency in distinguishing AI-generated hallucinations and impacting factors
Zhou, J., Zhang, J., Wan, R., Cui, X., Liu, Q., Guo, H., Shi, X., Fu, B., Meng, J., Yue, B., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06916-2 · Cited by 61
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53
Red teaming ChatGPT in medicine to yield real-world insights on model behavior
Chang, C. T., Farah, H., Gui, H., Rezaei, S. J., Bou-Khalil, C., Park, Y., Swaminathan, A., Omiye, J. A., Kolluri, A., Chaurasia, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01542-0 · Cited by 52
Self-citation
A survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 65
Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks
Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 49
Assessing and alleviating state anxiety in large language models
Ben‐Zion, Z., Witte, K., Jagadish, A. K., Duek, O. A., Harpaz‐Rotem, I., Khorsandian, M.-C., Burrer, A., Seifritz, E., Homan, P., Schulz, E., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01512-6 · Cited by 57
Implementing Large Language Models in Health Care: Clinician-Focused Review With Interactive Guideline
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 48
Artificial intelligence in pediatric healthcare: current applications, potential, and implementation considerations
Park, T., Lee, I., Lee, S. W., & Kong, S. W. (2025) · Korean Journal of Pediatrics · doi:10.3345/cep.2025.00962 · Cited by 44
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 37
A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains
Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02277-8 · Cited by 33
Benchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 38
Advantages and limitations of large language models for antibiotic prescribing and antimicrobial stewardship
Giacobbe, D. R., Marelli, C., Manna, B. L., Padua, D., Malva, A., Guastavino, S., Signori, A., Mora, S., Rosso, N., Campi, C., et al. (2025) · npj Antimicrobials and Resistance · doi:10.1038/s44259-025-00084-5 · Cited by 30
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Development and Evaluation of a Retrieval-Augmented Generation Chatbot for Orthopedic and Trauma Surgery Patient Education: Mixed-Methods Study
Baur, D., Ansorg, J., Heyde, C., & Voelker, A. (2025) · JMIR AI · doi:10.2196/75262 · Cited by 31
Beyond Assistance: The Case for Role Separation in AI-Human Radiology Workflows
Rajpurkar, P., & Topol, E. J. (2025) · Radiology · doi:10.1148/radiol.250477 · Cited by 26
Self-citation
How large language model-powered conversational agents influence decision making in domestic medical triage contexts
Gómez, C., Yin, J., Huang, C., & Unberath, M. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1427463 · Cited by 27
Reproducible generative artificial intelligence evaluation for health care: a clinician-in-the-loop approach
Livingston, L., Featherstone-Uwague, A., Barry, A., Barretto, K., Morey, T., Herrmannová, D., & Avula, V. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf054 · Cited by 26
Streamlining evidence based clinical recommendations with large language models
Li, D., Jiang, N., Huang, K., Tu, R., Ouyang, S., Yu, H., Lin, Q., Chen, Y., Zhou, T., Tong, D., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02273-y · Cited by 22
Large language models for frontline healthcare support in low-resource settings
Rutunda, S., Williams, G., Kabanda, K., Nkurunziza, F., Uwiduhaye, S., Rugegamanzi, E., Nshimiyimana, C., Menon, V., Emmanuel-Fabula, M., Denniston, A. K., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00038-1 · Cited by 20
Large Language and Vision Assistant in dermatology: a game changer or just hype?
Göktaş, P., Gülseren, D., & Tobin, A. (2024) · Clinical and Experimental Dermatology · doi:10.1093/ced/llae119 · Cited by 20
Synthetic Patient–Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation
Haider, S. A., Prabha, S., Gomez-Cabello, C. A., Borna, S., Genovese, A., Trabilsy, M., Collaço, B. G., Wood, N. G., Bagaria, S. P., Tao, C., et al. (2025) · Sensors · doi:10.3390/s25144305 · Cited by 22
When It Comes to Benchmarks, Humans Are the Only Way
Rodman, A. M., Zwaan, L., Olson, A. P. J., & Manrai, A. K. (2025) · NEJM AI · doi:10.1056/aie2500143 · Cited by 20
Artificial intelligence in primary care: innovation at a crossroads
Laranjo, L., Car, L. T., Payne, R. E., Neves, A. L., Kidd, M. R., & Miranda, J. J. (2025) · The Lancet Primary Care · doi:10.1016/j.lanprc.2025.100078 · Cited by 25
Principles to guide clinical AI readiness and move from benchmarks to real-world evaluation
Azad, T. D., Krumholz, H. M., & Saria, S. (2026) · Nature Medicine · doi:10.1038/s41591-025-04198-1 · Cited by 17
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A. M., Cai, T., Dagan, N., Balicer, R. D., Loscalzo, J., Kohane, I. S., & Žitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 22
DermaGPT a federated multimodal framework with a meta learned trust function for interpretable dermatology diagnostics
Hashjin, N. M., Amiri, M. H., & Najafabadi, M. K. (2026) · Scientific Reports · doi:10.1038/s41598-026-38715-0 · Cited by 18
Barriers and opportunities of scaling ambient AI scribes for clinical documentation across diverse healthcare settings
Ohde, J. W., Thompson, A., Liu, Z., Rost, L. M., Overgaard, J. D., Tan, J. Y. E., Overgaard, S. M., Sarmiento, R. F., Ke, Y., Liew, J. C. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02554-0 · Cited by 17
Reporting checklist for foundation and large language models in medical research (REFINE): an international consensus guideline
Mese, I., D’Antonoli, T. A., Bluethgen, C., Bressem, K., Cuocolo, R., Chaudhari, A., Tejani, A. S., Isaac, A., Ponsiglione, A., Meddeb, A., et al. (2026) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2026.263812 · Cited by 17
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 21
Understanding the robustness of vision-language models to medical image artefacts
Cheng, Z., Ong, A. Y., Wagner, S. K., Merle, D. A., Ju, L., Zhang, H., Chen, R., Pang, L., Li, B., He, T., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02108-w · Cited by 18
Robustness tests for biomedical foundation models should tailor to specifications
Xian, R. P., Baker, N., David, T., Cui, Q., Holmgren, A. J., Bauer, S., Sushil, M., & Abbasi-Asl, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01926-2 · Cited by 14
Ten Natural Language Processing Tasks with Generative Artificial Intelligence
Golec, J., & Hachaj, T. (2025) · Applied Sciences · doi:10.3390/app15169057 · Cited by 14
Legal Dimensions of AI Contracts in Sports Talent Management: Challenges and SolutionsPreprint
Asiabar, M. G., Asiabar, M. G., & Asiabar, A. G. (2025) · doi:10.14293/pr2199.001393.v1 · Cited by 13
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.266 · Cited by 13
Transforming breast cancer diagnosis and treatment with large language Models: A comprehensive survey.
Ghorbian, M., Ghobaei-Arani, M., & Ghorbian, S. (2025) · Methods · doi:10.1016/j.ymeth.2025.04.001 · Cited by 14
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 344
Independent and collaborative performance of large language models and healthcare professionals in diagnosis and triage
Chen, M., Wu, Y., Ma, J., Jia, X., Gao, C., Zhao, F., & Qiao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02409-8 · Cited by 15
Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility
Allison, D. M., & Herzog, S. (2025) · Risk Analysis · doi:10.1111/risa.70105 · Cited by 10
Quality safety and disparity of an AI chatbot in managing chronic diseases: simulated patient experiments
Si, Y., Meng, Y., Chen, X., An, R., Mao, L., Li, B., Bateman, H., Zhang, H., Fan, H., Zu, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01956-w · Cited by 9
AI and Digital Tools in Dermatology: Addressing Access and Misinformation
du Crest, D., Madhumita, M., Enbiale, W., Postigo, J. A. R., Malvehy, J., Wongvibulsin, S., Gupta, S., Kittler, H., Skayem, C., Mahto, A., et al. (2026) · JMIR Dermatology · doi:10.2196/79044 · Cited by 9
Self-citation
Multimodal Vision–Language Models in Medical Imaging: A Survey of Retrieval, Interpretability, and Trust
Imran, M., & Lee, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3656368 · Cited by 11
A clinical environment simulator for dynamic AI evaluation
Luo, L., Kim, S. E., Zhang, X., Kernbach, J. M., Kenia, R., Acosta, J. N., Nathanson, L. A., Haimovich, A. D., Rodman, A. M., Goh, E., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04252-6 · Cited by 14
Self-citation
Clinical Use of Non-Certified Generative AI in Healthcare: Governing the Regulatory Grey Zone from Convenience to Legal Accountability
Sirago, G., Calò, F., Vinci, A., Visci, P., Solarino, B., Dell'Erba, A. S., & Ferorelli, D. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02363-8 · Cited by 9
Zero-Shot Large Language Models for Long Clinical Text Summarization with Temporal ReasoningPreprint
Kruse, M., Hu, S., Derby, N., Wu, Y., Stonbraker, S., Yao, B., Wang, D., Goldberg, E., & Gao, Y. (2025) · medRxiv · doi:10.1101/2025.07.21.25331947 · Cited by 9
A Patient-in-the-Loop Approach to Artificial Intelligence in Medicine
Griot, M., & Walker, G. A. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.14460 · Cited by 8
The Financial, Operational, and Clinical Advantages of Generalist Radiology AI
Dogra, S., Zhang, X., Silva, E., & Rajpurkar, P. (2025) · Radiology · doi:10.1148/radiol.242362 · Cited by 8
Self-citation
Will Autonomous AI Exceed AI-Aided Physicians as the Best Medical Care?
Emanuel, E., Baker-Butler, A., Khosla, N., & Khosla, V. (2026) · JAMA · doi:10.1001/jama.2026.15380 · Cited by 8
Using Large Language Models to Simulate History Taking: Implications for Symptom-Based Medical Education
Huh, C. Y., Lee, J., Kim, G., Jang, Y., Ko, H.-S., Suh, M. J., Hwang, S., Son, H. J., Song, J., Kim, S., et al. (2025) · Information · doi:10.3390/info16080653 · Cited by 7
Foundation models in clinical oncology: Progresses and perspectives
Li, J., Fu, J., Geng, X., & Wang, H. (2025) · Cancer Letters · doi:10.1016/j.canlet.2025.218220 · Cited by 8
Large Language Models in Qualitative Analysis: Comparing Traditional and Researcher-Interpreted Approaches
Misra, R., Dahal, R., Kirk, B., Khan, R., Dogan, G., Chataut, R., & Gyawali, P. (2026) · International Journal of Qualitative Methods · doi:10.1177/16094069261426100 · Cited by 7
Artificial intelligence–enabled precision medicine for inflammatory skin diseases
Tang, A. S., Wei, M. L., Haemel, A., La, C., Sirota, M., & Lee, E. Y. (2026) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2025.10.596 · Cited by 10
ChatSOS: Vector database augmented generative question answering assistant in safety engineering
Tang, H., Chen, D., Chu, Q., & Liu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128644 · Cited by 7
Current Landscape and Future Directions Regarding Generative Large Language Models in Stroke Care: Scoping Review
Zhu, X., Dai, W., Evans, R. D., Geng, X., Mu, A., & Liu, Z. (2025) · JMIR Medical Informatics · doi:10.2196/76636 · Cited by 6
A bibliometric analysis of artificial intelligence in medical education (2015–2025)
Cheng, W., Hu, Z., & Yu, H. (2025) · Medicine · doi:10.1097/md.0000000000045684 · Cited by 6
Artificial intelligence in medicine: a position paper by the Italian Society of Internal Medicine
Balsano, C., Cabitza, F., Cicco, S., Gori, M., Malerba, D., Montagna, M., Tarquini, R., Vacca, A., & The Working Group on Artificial Intelligence, D. T. O. T. I. S. O. I. M. (. (2025) · Internal and Emergency Medicine · doi:10.1007/s11739-025-04146-4 · Cited by 7
A multi-expert deep learning framework with LLM-guided arbitration for multimodal histopathology prediction
Debsarkar, S. S., & Prasath, V. B. S. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102704 · Cited by 6
Application of Artificial Intelligence in Education of Prosthodontics and Implant Dentistry: A Review
Gan, Y., Guo, J., Zhai, J., & Huang, J. (2026) · Advances in Medical Education and Practice · doi:10.2147/amep.s573041 · Cited by 6
Patients Facing Large Language Models in Oncology: A Narrative Review
Raynaud, C., Wu, D., Levy, J., Marengo, M., & Bibault, J. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00149 · Cited by 6
MAP: evaluation and multi-agent enhancement of large language models for inpatient pathways
Chen, Z., Peng, Z., Liang, X., Wang, C., Liang, P., Zeng, L., Ju, M., & Yuan, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00085-0 · Cited by 17
Multi-center benchmarking of large language models for clinical decision support in lung cancer screening
Duan, Z., Huang, X., Lu, R., Xu, W., Liu, H., Geng, Y., Takahashi, N., Wu, Y., Wang, Q., Song, Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102465 · Cited by 5
Rethinking AI Workflows: Guidelines for Scientific Evaluation in Digital Health Companies
McAlister, K. L., Gonzales, L., & Huberty, J. (2025) · JMIR AI · doi:10.2196/71798 · Cited by 7
Evaluating the Insights of ChatGPT, Gemini and Expert Surgeons in Revision Rhinoplasty Consultation
Bastaninejad, S., Alipour, S., Ishida, L. C., Mohebbi, M., Mousavi-Asl, B., Heidari, F., & Azimi, H. (2025) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05464-w · Cited by 5
Embodied conversational agents for collecting patient-reported data: effective for some, challenging for others – results of a cross-sectional study in multiple sclerosis care
Weimann, T. G., Schlieter, H., Konofalska, U., & Ziemssen, T. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2641598 · Cited by 5
Evaluating GPT-4’s visual interpretation and clinical reasoning on emergency settings: A 5-year analysis
Wang, T.-H., Jheng, J.-C., Tseng, Y.-T., Chen, L., & Chen, Y. (2025) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000001273 · Cited by 5
Performance of o1 pro and GPT-4 in Self-Assessment Questions for Nephrology Board Renewal
Noda, R., Yuasa, C., Kitano, F., Ichikawa, D., & Shibagaki, Y. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1702668 · Cited by 5
Artificial intelligence-enabled precision medicine for inflammatory skin diseasesPreprint
Tang, A., Wei, M. L., Haemel, A. K., La, C., Sirota, M., & Lee, E. Y. (2025) · PubMed · doi:10.48550/arxiv.2505.09527 · Cited by 5
Generative AI in healthcare: challenges to patient agency and ethical implications
Holmes, S. A., Faria, V., & Moulton, E. A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1524553 · Cited by 4
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4
A multicenter multifunctional assessment of large language models in pure-tone audiogram interpretation for patients
Liang, J., Xing, M., Xiang, P., Wang, G., Chen, M., Fang, Q., Zhou, T., Lu, Z., Leng, X., Huang, J., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02537-1 · Cited by 4
Automating evaluation of LLM-generated responses to patient questions about rare diseases
Zhao, M., Oh, I. Y., Gupta, A., Cohen‐Cutler, S., Harmoney, K. M., Lai, A. M., & Sisk, B. A. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag054 · Cited by 4
The patient-safety implications of AI-based communication with migrants in general practice: a scoping review
Cronin, A., Kelly, A., Wrona, M., O'Donnell, P., Hassan, A. A., Myles, T., Fallon, T., & MacFarlane, A. E. (2025) · BJGP Open · doi:10.3399/bjgpo.2025.0107 · Cited by 5
A large language model for clinical outcome adjudication from telephone follow-up interviews: a secondary analysis of a multicenter randomized clinical trial
Shi, Z., Wu, B., Hu, B., Zhong, J., Li, Z., Zhang, F., Chen, Z., Yang, C., Guo, B., Xu, Q., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66910-6 · Cited by 4
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 7
A S.C.O.R.E. framework for evaluating open-ended responses from large language models in healthcare
Tan, T. F., Elangovan, K., ONG, J., Ke, Y., Lee, A., Shah, N. H., Sung, J., Wong, T. Y., Xue, L., Liu, N., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102883 · Cited by 4
“The Machine Will See You Now”: A Clinician's Perspective on Artificial “Intelligence” In Clinical Care
Mahajan, A., & Lees, A. J. (2025) · Movement Disorders Clinical Practice · doi:10.1002/mdc3.70054 · Cited by 4
The application and challenges of ChatGPT in laboratory medicine
Niu, Z., Kuang, X., Chen, J., Cai, X., & Zhang, P. (2025) · Advances in Laboratory Medicine / Avances en Medicina de Laboratorio · doi:10.1515/almed-2025-0080 · Cited by 3
New chapter in pediatric medicine: technological evolution, application, and evaluation system of large language models
Zhu, S., Xie, Y., Tang, Y., Zhou, Y., Zhao, R., & Dong, X. (2025) · European Journal of Pediatrics · doi:10.1007/s00431-025-06602-x · Cited by 4
Applications of artificial intelligence and large language models in cancer immunotherapy
Su, Y., Liu, J., Li, J., Huang, X., Hajdú, A., Fu, R., Xu, B., Wang, Y., Liu, X., Li, Z., et al. (2025) · Current Proteomics · doi:10.1016/j.curpro.2025.100057 · Cited by 3
Grounding large language models in clinical diagnostics
Chen, X., Zhou, H., Yi, H., You, M., Liu, W., Li, W., Qin, Z., Li, H., Zhang, X., Guo, Y., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70274-w · Cited by 9
Self-citation
Multi-Axial Analysis of Clinical Reasoning in Large Language Models: Inter-Verifier Disagreement and Its Implications for Automated Evaluation
Byun, H., Lee, D., Jung, M., & Jang, B. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02440-y · Cited by 3
A Platform for Interactive AI Character Experiences
Wampfler, R., Yang, C., Elste, D., Kovačević, N., Witzig, P., & Groß, M. H. (2025) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3721238.3730762 · Cited by 4
Evaluating Large Reasoning Model Performance on Complex Medical Scenarios In The MMLU-Pro BenchmarkPreprint
Hoyt, R. E., Knight, D. R. T., Haider, M., & Bajwa, M. (2025) · medRxiv · doi:10.1101/2025.04.07.25325385 · Cited by 3
Comparative Performance of agentic AI and Physicians in Taking Clinical History across Leading Large Language Models (LLMs)Preprint
Steinbuch, S., de Vos‐Hillebrand, L., O'Neill-Dee, C., Wu, I., Steinbuch, H., Kulcsár, Z., Ranjan, A., Verma, A., Landsberg, J., Dietrich, D., et al. (2026) · medRxiv · doi:10.64898/2026.01.23.26344723 · Cited by 3
Auditing medical multi-agent AI reveals risks of false consensus
Zhu, Y., Gu, L., Wang, Z., Sang, H., Sui, D., Tang, W., Mi, L., Wang, Y., Gao, J., Yao, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.10185 · Cited by 17
Tailoring Treatment in the Age of AI: A Systematic Review of Large Language Models in Personalized Healthcare
Souza, G. D. P., Melo, G., & Schneider, D. (2025) · Informatics · doi:10.3390/informatics12040113 · Cited by 2

Authors: Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., Daneshjou, R., & Rajpurkar, P.
Affiliations: Harvard University (US)
Washington Hospital (US)
Northwestern University (US)
University of California, Los Angeles (US)
Stanford University (US)
Stanford Medicine (US)
Corresponding author: Daneshjou, Roxana
Rajpurkar, Pranav
ORCID: Johri, Shreya
Tran, Benjamin A.
Schlessinger, Daniel I.
Wongvibulsin, Shannon
Barnes, Leandra A.
Zhou, Hong-Yu
Cai, Zhuo Ran
Van Allen, Eliezer M.
Kim, David
Daneshjou, Roxana
Rajpurkar, Pranav
Venue: Nature Medicine
Publisher: Springer Science and Business Media LLC
Published: 2025-01
Volume: 31
Issue: 1
Pages: 77-86
ISSN: 1078-8956 (ISSN-L)
1546-170X
DOI: 10.1038/s41591-024-03328-5
OpenAlex: W4405995927
Semantic Scholar: 597f3cc7c41ba1da58624f79771b26a5fd8de1ba
Cited by: 201
References: 45
Citations by year: 2026: 114 · 2025: 86
FWCI: 47.80
Type: journal article
Language: en
Open access: closed
Licence: www.springernature.com/gp/researchers/text-and-data-mining (tdm)
www.springernature.com/gp/researchers/text-and-data-mining (vor)
Locations: Nature Medicine (publishedVersion)
PubMed (publishedVersion)
Topics: Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords: large language models, clinical evaluation, doctor-patient interaction, medical history, diagnostic accuracy, GPT-4, GPT-3.5, Mistral, Llama 2, GPT-4V, multimodal assessment, evaluation framework
MeSH terms: Large Language Models, Artificial Intelligence, Communication, Humans, Medical History Taking, Physician-Patient Relations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

An evaluation framework for clinical use of large language models in patient interaction tasks

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks