Evaluating large language models and agents in healthcare: key challenges in clinical applications
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have emerged as transformative tools with significant potential across healthcare and medicine. In clinical settings, they hold promises for tasks ranging from clinical decision support to patient education. Advances in LLM agents further broaden their utility by enabling multimodal processing and multitask handling in complex clinical workflows. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the high-risk nature of healthcare and the complexity of medical data. This paper provides a comprehensive overview of current evaluation practices for LLMs and LLM agents in medicine. We contributed 3 main aspects: First, we summarized data sources used in evaluations, including existing medical resources and manually designed clinical questions, offering a basis for LLM evaluation in medical settings. Second, we analyzed key medical task scenarios: closed-ended tasks, open-ended tasks, image processing tasks, and real-world multitask scenarios involving LLM agents, thereby offering guidance for further research across different medical applications. Third, we compared evaluation methods and dimensions, covering both automated metrics and human expert assessments, while addressing traditional accuracy measures alongside agent-specific dimensions, such as tool usage and reasoning capabilities. Finally, we identified key challenges and opportunities in this evolving field, emphasizing the need for continued research and interdisciplinary collaboration between healthcare professionals and computer scientists to ensure safe, ethical, and effective deployment of LLMs in clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citationAyers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
The Medical Segmentation Decathlon
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30695-9 · Cited by 1,384
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738
Self-citationChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
Research works printed but not linked: 70
- Ozkaya, 2023, Application of large language models to software engineering tasks: opportunities, risks, and implications, IEEE Softw, 40, 4, 10.1109/MS.2023.3248401 · DOI · Paper page
- Alqahtani, 2023, The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research, Res Social Adm Pharm, 19, 1236, 10.1016/j.sapharm.2023.05.016 · DOI · Paper page
- Thirunavukarasu, 2023, Large language models in medicine, Nat Med, 29, 1930, 10.1038/s41591-023-02448-8 · DOI · Paper page
- Demszky, 2023, Using large language models in psychology, Nat Rev Psychol, 10.1038/s44159-023-00241-5 · DOI · Paper page
- Zhang, 2024, Examining the role of large language models in orthopedics: systematic review, J Med Internet Res, 26, 10.2196/59607 · DOI
- Chen, 2024, EyeGPT for patient inquiries and medical education: development and validation of an ophthalmology large language model, J Med Internet Res, 26, 10.2196/60063 · DOI
- Zhang H, Chen J, Jiang F, et al. HuatuoGPT, towards taming language model to be a doctor. 2023. doi:10.48550/ARXIV.2305.15075. · DOI · Paper page
- Mihalache, 2024, Accuracy of an artificial intelligence Chatbot’s interpretation of clinical ophthalmic images, JAMA Ophthalmol, 142, 321, 10.1001/jamaophthalmol.2024.0017 · DOI
- Wang, 2023, Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI, Int J Med Inform, 177, 10.1016/j.ijmedinf.2023.105173 · DOI
- Antaki, 2023, Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings, Ophthalmol Sci, 3, 10.1016/j.xops.2023.100324 · DOI · Paper page
- Ali, 2023, Performance of ChatGPT and GPT-4 on neurosurgery written board examinations, Neurosurgery, 93, 1353, 10.1227/neu.0000000000002632 · DOI · Paper page
- Long, 2024, A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study, JMIR Med Educ, 10, 10.2196/49970 · DOI · Paper page
- Beşler, 2024, Evaluating GPT-4o’s performance in the official European Board of Radiology exam: a Comprehensive Assessment, Acad Radiol, 31, 4365, 10.1016/j.acra.2024.09.005 · DOI
- Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education in prostate cancer: Multimetric assessment, J Med Internet Res, 26, 10.2196/55939 · DOI · Paper page
- Mukherjee, 2024, Evaluation of GPT large language model performance on RSNA 2023 case of the day questions, Radiology, 313, 10.1148/radiol.240609 · DOI
- Kim, 2024, ChatGPT vision for radiological interpretation: an investigation using medical school radiology examinations, Korean J Radiol, 25, 403, 10.3348/kjr.2024.0017 · DOI
- Khan, 2021, A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability, Lancet Digit Health, 3, e7
- Hurley, 2024, Evaluation of high-quality information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery, Arthroscopy, 40, 726, 10.1016/j.arthro.2023.07.048 · DOI · Paper page
- Onder, 2024, Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy, Sci Rep, 14, 243, 10.1038/s41598-023-50884-w · DOI
- Marshall, 2024, Investigating the accuracy and completeness of an artificial intelligence large language model about uveitis: an evaluation of ChatGPT, Ocul Immunol Inflamm, 32, 2052, 10.1080/09273948.2024.2317417 · DOI
- Li W, Li L, Xiang T, et al. Can multiple-choice questions really be useful in detecting the abilities of LLMs? 2024. doi:10.48550/ARXIV.2403.17752. · arXiv · doi:10.48550/arxiv.2403.17752
- Cheng, 2023, Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study, J Med Internet Res, 25, 10.2196/51229 · DOI · Paper page
- Hake, 2024, Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts, Ann Fam Med, 22, 113, 10.1370/afm.3075 · DOI · Paper page
- Caterson, 2024, Application of generative language models to orthopaedic practice, BMJ Open, 14, 10.1136/bmjopen-2023-076484 · DOI
- Chung, 2023, Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients, Digital Health, 9, 10.1177/20552076231221620 · DOI · Paper page
- Chen, 2023, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics, 39, btad557, 10.1093/bioinformatics/btad557 · DOI · Paper page
- Jahan, 2024, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Comput Biol Med, 171, 10.1016/j.compbiomed.2024.108189 · DOI · Paper page
- Gu, 2024, Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model, Artif Intell Med, 150, 10.1016/j.artmed.2024.102822 · DOI
- Cinquin, 2024, ChIP-GPT: a managed large language model for robust data extraction from biomedical database records, Brief Bioinform, 25, bbad535, 10.1093/bib/bbad535 · DOI
- Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
- Ge, 2024, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology, 80, 1158, 10.1097/HEP.0000000000000834 · DOI · Paper page
- Zhu, 2023, Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?, J Transl Med, 21, 269, 10.1186/s12967-023-04123-5 · DOI
- Huang, 2024, Assessment of a large language model’s responses to questions and cases about glaucoma and retina management, JAMA Ophthalmol, 142, 393, 10.1001/jamaophthalmol.2023.6917 · DOI · Paper page
- Civettini, 2024, Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making, Br J Haematol, 204, 1523, 10.1111/bjh.19200 · DOI
- Sorin, 2023, Large language model (ChatGPT) as a support tool for breast tumor board, NPJ Breast Cancer, 9, 44, 10.1038/s41523-023-00557-8 · DOI · Paper page
- Ali, 2023, ChatGPT and lacrimal drainage disorders: performance and scope of improvement, Ophthalmic Plast Reconstr Surg, 39, 221, 10.1097/IOP.0000000000002418 · DOI · Paper page
- Potapenko, 2023, Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT, Acta Ophthalmol, 101, 829, 10.1111/aos.15661 · DOI
- Rasmussen, 2023, Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis, Graefes Arch Clin Exp Ophthalmol, 261, 3041, 10.1007/s00417-023-06078-1 · DOI
- Wilhelm, 2023, Large Language models for therapy recommendations across 3 clinical specialties: comparative study, J Med Internet Res, 25, 10.2196/49324 · DOI · Paper page
- Uz, 2023, Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases?, Int J Rheum Dis, 26, 1343, 10.1111/1756-185X.14749 · DOI
- Cao, 2023, Accuracy of information provided by ChatGPT regarding liver cancer surveillance and diagnosis, AJR Am J Roentgenol, 221, 556, 10.2214/AJR.23.29493 · DOI · Paper page
- Balel, 2023, Can ChatGPT be used in oral and maxillofacial surgery?, J Stomatol Oral Maxillofac Surg, 124, 10.1016/j.jormas.2023.101471 · DOI · Paper page
- Munoz-Zuluaga, 2023, Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine, Clin Chem, 69, 939, 10.1093/clinchem/hvad058 · DOI · Paper page
- Young, 2023, The utility of ChatGPT in generating patient-facing and clinical responses for melanoma, J Am Acad Dermatol, 89, 602, 10.1016/j.jaad.2023.05.024 · DOI
- Cadamuro, 2023, Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results, Clin Chem Lab Med, 61, 1158, 10.1515/cclm-2023-0355 · DOI · Paper page
- Wan, 2024, Med-unic: unifying cross-lingual medical vision-language pre-training by diminishing bias, Adv Neural Inf Process Syst, 36
- Liu, 2024, A systematic evaluation of GPT-4V’s multimodal capability for chest X-ray image analysis, Meta-Radiology, 2, 10.1016/j.metrad.2024.100099 · DOI
- Wang, 2023, R2gengpt: radiology report generation with frozen llms, Meta-Radiology, 1, 10.1016/j.metrad.2023.100033 · DOI · Paper page
- Lin, 2023, Medical visual question answering: a survey, Artif Intell Med, 143, 10.1016/j.artmed.2023.102611 · DOI · Paper page
- Li, 2023, A comprehensive sstudy of GPT-4V’s multimodal capabilities in medical imaging, medRxiv, 2023
- Gu Z, Yin C, Liu F, et al. MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. 2024. doi:10.48550/ARXIV.2407.02730. · DOI
- Wu, 2024, Hallucination benchmark in medical visual question answering
- Oh, 2025, LLM-driven multimodal target volume contouring in radiation oncology, Nat Commun, 16, 718, 10.1038/s41467-025-55963-2 · doi:10.1038/s41467-025-55963-2
- Harrer, 2024, Generative AI agents are transforming biology research: high resolution functional genome annotation for multiscale understanding of life, EBioMedicine, 109, 10.1016/j.ebiom.2024.105446 · DOI
- Wang, 2025, A survey of LLM-based agents in medicine: how far are we from Baymax?
- Liu, 2024, AgentBench: evaluating LLMs as agents
- Zheng, 2024, Can modern LLMs act as agent cores in radiology environments?
- Rjoob, 2022, Machine learning and the electrocardiogram over two decades: Time series and meta-analysis of the algorithms, evaluation metrics and applications, Artif Intell Med, 132, 10.1016/j.artmed.2022.102381 · DOI
- Chen, 2024, ChatFFA: an ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography, iScience, 27, 10.1016/j.isci.2024.110021 · DOI
- Zhao, 2024, Slit lamp report generation and question answering: development and validation of a multimodal transformer model with large language model integration, J Med Internet Res, 26, 10.2196/54047 · DOI
- Charnock, 1999, DISCERN: an instrument for judging the quality of written consumer health information on treatment choices, J Epidemiol Community Health, 53, 105, 10.1136/jech.53.2.105 · DOI · Paper page
- Silberg, 1997, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware, JAMA, 277, 1244, 10.1001/jama.1997.03540390074039 · DOI · Paper page
- Ozduran, 2022, Evaluating the readability, quality and reliability of online patient education materials on post-covid pain, PeerJ, 10, 10.7717/peerj.13686 · DOI
- Rensis, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 5
- Wang, 2024, Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: comparative study, J Med Internet Res, 26, 10.2196/58041 · DOI · Paper page
- Rau, 2023, A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines, Radiology, 308, 10.1148/radiol.230970 · DOI · Paper page
- Xu, 2025, DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning
- Resnikoff, 2020, Estimated number of ophthalmologists worldwide (International Council of Ophthalmology update): will we meet the needs?, Br J Ophthalmol, 104, 588, 10.1136/bjophthalmol-2019-314336 · DOI · Paper page
- Yang SH, Zhao HJ, Zhu GY, et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2024AAAI 2024;38(17):19368-19376. doi: 10.1609/aaai.v38i17.29907. · DOI · Paper page
- Li, 2024, Integrated image-based deep learning and language models for primary diabetes care, Nat Med, 30, 2886, 10.1038/s41591-024-03139-8 · DOI · Paper page
Cited by
From language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50
“…workflows where they perform planning, decision making, and tool interaction in various real-world applications, including research assistance [6], software development [7], drug discoveries [8], multi-robot systems [9], clinical support [10], game simulation [11] and scientific simulations [12].”
Tzanis, E., Adams, L. C., Akinci D’Antonoli, T., Bressem, K. K., Cuocolo, R., Kocak, B., Malamateniou, C., & Klontzas, M. E. (2026) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2025.10.002 · Cited by 16
Jiang, X.-H., Yuan, X.-H., Zhao, H., & Peng, J.-S. (2025) · Journal of Medical Internet Research · doi:10.2196/76968 · Cited by 10
Agentic artificial intelligence is the future of cancer detection and diagnosis
Rahman, S., Hosain, M. T., Fahad, N., Morol, M. K., & Hossen, M. J. (2026) · Array · doi:10.1016/j.array.2025.100676 · Cited by 4
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
“However, prior healthcare multi-agent work largely centers on deliberation under the assumption that key evidence is already in-context, leaving evidence acquisition unsystematic (Chen et al., 2025; Gorenshtein et al., 2025; Wang et al., 2025).”
Metadata
- Authors
- Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D.
- Venue
- Intelligent Medicine
- Publisher
- Elsevier BV
- Published
- 2025-05
- Volume
- 5
- Issue
- 2
- Pages
- 151-163
- ISSN
- 2096-9376 (ISSN-L)
- 2667-1026
- OpenAlex
- W4409091111
- Cited by
- 68
- References
- 62
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Funders
- The Hong Kong Polytechnic University
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating large language models and agents in healthcare: key challenges in clinical applications. https://app.cochat.ai/papers/10.1016/j.imed.2025.03.002
- The paper
- Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025). Evaluating large language models and agents in healthcare: key challenges in clinical applications. Intelligent Medicine. https://doi.org/10.1016/j.imed.2025.03.002
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.