The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
An evaluation framework for clinical use of large language models in patient interaction tasks
Abstract
The integration of large language models (LLMs) into clinical diagnostics has the potential to transform doctor-patient interactions. However, the readiness of these models for real-world clinical application remains inadequately tested. This paper introduces the Conversational Reasoning Assessment Framework for Testing in Medicine (CRAFT-MD) approach for evaluating clinical LLMs. Unlike traditional methods that rely on structured medical examinations, CRAFT-MD focuses on natural dialogues, using simulated artificial intelligence agents to interact with LLMs in a controlled environment. We applied CRAFT-MD to assess the diagnostic capabilities of GPT-4, GPT-3.5, Mistral and LLaMA-2-7b across 12 medical specialties. Our experiments revealed critical insights into the limitations of current LLMs in terms of clinical conversational reasoning, history-taking and diagnostic accuracy. These limitations also persisted when analyzing multimodal conversational and visual assessment capabilities of GPT-4V. We propose a comprehensive set of recommendations for future evaluations of clinical LLMs based on our empirical findings. These recommendations emphasize realistic doctor-patient conversations, comprehensive history-taking, open-ended questioning and using a combination of automated and expert evaluations. The introduction of CRAFT-MD marks an advancement in testing of clinical LLMs, aiming to ensure that these models augment medical practice effectively and ethically.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous benchmarks like AMIE23, SAPS30, and CRAFT-MD31 focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [31]
In the citing paper
“Although STARD-AI offers a strong foundation for transparent reporting, complementary frameworks such as CRAFT-MD may be better suited for evaluating different forms of AI-driven clinical support 53 .”
From The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence (2025) · cites this paper as [53]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Self-citationAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Self-citation
Research works printed but not linked: 1
- Image Challenge. N. Engl. J. Med. https://www.nejm.org/image-challenge (2024).
Cited by
Sorted by citations
Items 201–224 of 224
Language models reflect clinical evidence but fail to adapt it to patientsPreprint
He, S., Joseph, J. W., Safari, P., Goff, A. C., Slusarz, P. J., Mohamed, A., Lord, S., Goldstein, J. N., & Raja, A. S. (2026) · medRxiv · doi:10.64898/2026.09.14.26363047
Avoiding common failures in AI for health and medicine
Salaudeen, O., Zhang, H., Kim, E., Singh, K., & Ghassemi, M. (2026) · Cell · doi:10.1016/j.cell.2026.07.025
Sorka, M., Gorenshtein, A., Abramovitch, H., Soontrapa, P., Shelly, S., & Aran, D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag143
Stummer, F. O., Tsai, T. K., Wikberg, C., Nymberg, V. M., Radlgruber, I., Müller, A., Riegler, M. A., Petrazzuoli, F., Vinker, S., & Frese, T. (2026) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2026.2735209
Large language models as integrative intelligence for multimodal cardiovascular decision support
Mitrovic, P. (2026) · Journal of Clinical and Translational Research · doi:10.36922/jctr026290071
Large language models in perioperative anesthesia: a risk-based roadmap for safe clinical adoption
Tan, Z., He, Q., Li, Q., & Li, X. (2026) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-026-00196-9
Collaço, B. G., Wood, N. G., Yu, Y., Malena, C. R., Srinivasagam, P., Boon, A. L., Fang, Z., Bhagra, A., Leibovich, B. C., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13101115
Üçdal, M. T., Yurtsever, K., Yıldız, P., Akalın, A., Mert, K. U., & Guven, G. S. (2026) · Research Square · doi:10.21203/rs.3.rs-10682358/v1
Yin, J., Xiao, X., Ke, S., & Wang, Y. (2026) · Digital Health · doi:10.1177/20552076261493714
Performance of large language models in prescription review: a comparative study
Huang, C., Sun, Y., & Liu, W. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1939786
Evaluating Symptom Fidelity in LLM-Driven Simulated Patients Using PHQ-9 and GAD-7
Opachalearn, P., Wongpatikaseree, K., & Kertkeidkachorn, N. (2026) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr70517.2026.00028
Entity-centric evaluation of large language model responses for medical question-answering tasks
Liu, Y., & Kolachalama, V. B. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001752 · Cited by 1
MedNLI-Pain benchmarks a minimally necessary requirement for large language model alignment
Wong, C., Kumar, D., Muse, A., Wang, P., Wintz, F., East, S.-A., Lyou, J., Yanamala, N. V. K., & Wood-Doughty, Z. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag143
Zhang, J., Ma, Y., Zhang, R., Su, R., & Ma, K. (2026) · Research Square · doi:10.21203/rs.3.rs-10798645/v1
Yang, J. (2025) · International Conference on Mechanical Automation and Engineering Applications (ICMAEA) · doi:10.1109/icmaea64564.2025.11725548
Large Language Models in Hearing CarePreprint
Wang, S., Mo, C., Wu¹, J., Siu, A. K. Y., Zhao, F., Wasmann, J.-W. A., Swanepoel, D. W., & Manchaiah, V. K. C. (2026) · Research Square · doi:10.21203/rs.3.rs-11384026/v1
Clinical Tasks Explain AI Readiness Across Medical SpecialtiesPreprint
Meskó, B., Katonai, G., Chen, S., Sheng, B., Wong, T. Y., & Topol, E. (2026) · Research Square · doi:10.21203/rs.3.rs-11164578/v1
Robustness Gap of Large Language Models in Nephrology
Soejima, A., Kitano, F., Ichikawa, D., Shibagaki, Y., & Noda, R. (2026) · Journal of Nephrology · doi:10.1093/joneph/aajag265
Oloko-Oba, M., Esenogho, E., Aruleba, K., & Cameron, M. (2026) · AI · doi:10.3390/ai7110439
Infrastructure, Not Benchmarks, Limits Autonomous AI in Primary Care (Preprint)Preprint
Nguyen, A., & Nguyen, V. G. (2026) · doi:10.2196/preprints.114754
Cornelisse, E., Kousa, D., Menz, B. D., Murphy-Reuter, B., Shahnam, A., Chin-Yee, B. H., Calabro, R., Li, L. X., Ramsey, I., Abuhelwa, A. Y., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/100531 · Cited by 1
Agentic AI Systems in Radiology: Design, Applications, Evaluation, and Challenges
Blüthgen, C., Veen, D. V., Truhn, D., Kather, J. N., Moor, M., Polacin, M., Chaudhari, A. S., Frauenfelder, T., Langlotz, C. P., Krauthammer, M. O., et al. (2026) · Radiology · doi:10.1148/radiol.253226 · Cited by 1
What young adults with persistent musculoskeletal pain need from generative AI between visitsPreprint
Pang, J., Li, Z., Xu, C., & Ge, C. (2026) · Research Square · doi:10.21203/rs.3.rs-11494498/v1
Geleceğin Hemşireliği: Yapay Zekâ ve Dijital Sağlık Ekosisteminde Dönüşüm
Doğan, S. E. (2026) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub1577.c6489
Metadata
- Authors
- Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., Daneshjou, R., & Rajpurkar, P.
- Affiliations
- Harvard University (US)
- Washington Hospital (US)
- Northwestern University (US)
- University of California, Los Angeles (US)
- Stanford University (US)
- Stanford Medicine (US)
- Corresponding author
- Daneshjou, Roxana
- Rajpurkar, Pranav
- ORCID
- Venue
- Nature Medicine
- Published
- 2025-01
- Volume
- 31
- Issue
- 1
- Pages
- 77-86
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4405995927
- Semantic Scholar
- 597f3cc7c41ba1da58624f79771b26a5fd8de1ba
- Cited by
- 201
- References
- 45
- Citations by year
- 2026: 114 · 2025: 86
- FWCI
- 47.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, clinical evaluation, doctor-patient interaction, medical history, diagnostic accuracy, GPT-4, GPT-3.5, Mistral, Llama 2, GPT-4V, multimodal assessment, evaluation framework
- MeSH terms
- Large Language Models, Artificial Intelligence, Communication, Humans, Medical History Taking, Physician-Patient Relations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
