CoChat

Evaluating large language models and agents in healthcare: key challenges in clinical applications

Open access
Chen, Xiaolan, Xiang, Jiayang, Lu, Shanfu, Liu, Yexin, He, Mingguang, Shi, Danli · 2025 · Intelligent Medicine
Cited by 68

The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have emerged as transformative tools with significant potential across healthcare and medicine. In clinical settings, they hold promises for tasks ranging from clinical decision support to patient education. Advances in LLM agents further broaden their utility by enabling multimodal processing and multitask handling in complex clinical workflows. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the high-risk nature of healthcare and the complexity of medical data. This paper provides a comprehensive overview of current evaluation practices for LLMs and LLM agents in medicine. We contributed 3 main aspects: First, we summarized data sources used in evaluations, including existing medical resources and manually designed clinical questions, offering a basis for LLM evaluation in medical settings. Second, we analyzed key medical task scenarios: closed-ended tasks, open-ended tasks, image processing tasks, and real-world multitask scenarios involving LLM agents, thereby offering guidance for further research across different medical applications. Third, we compared evaluation methods and dimensions, covering both automated metrics and human expert assessments, while addressing traditional accuracy measures alongside agent-specific dimensions, such as tool usage and reasoning capabilities. Finally, we identified key challenges and opportunities in this evolving field, emphasizing the need for continued research and interdisciplinary collaboration between healthcare professionals and computer scientists to ensure safe, ethical, and effective deployment of LLMs in clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.imed.2025.03.002

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 66

  1. Ozkaya, 2023, Application of large language models to software engineering tasks: opportunities, risks, and implications, IEEE Softw, 40, 4, 10.1109/MS.2023.3248401 · DOI · Paper page
  2. Alqahtani, 2023, The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research, Res Social Adm Pharm, 19, 1236, 10.1016/j.sapharm.2023.05.016 · DOI · Paper page
  3. Demszky, 2023, Using large language models in psychology, Nat Rev Psychol, 10.1038/s44159-023-00241-5 · DOI · Paper page
  4. Zhang, 2024, Examining the role of large language models in orthopedics: systematic review, J Med Internet Res, 26, 10.2196/59607 · DOI
  5. Chen, 2024, EyeGPT for patient inquiries and medical education: development and validation of an ophthalmology large language model, J Med Internet Res, 26, 10.2196/60063 · DOI
  6. Zhang H, Chen J, Jiang F, et al. HuatuoGPT, towards taming language model to be a doctor. 2023. doi:10.48550/ARXIV.2305.15075. · DOI · Paper page
  7. Mihalache, 2024, Accuracy of an artificial intelligence Chatbot’s interpretation of clinical ophthalmic images, JAMA Ophthalmol, 142, 321, 10.1001/jamaophthalmol.2024.0017 · DOI
  8. Wang, 2023, Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI, Int J Med Inform, 177, 10.1016/j.ijmedinf.2023.105173 · DOI · Paper page
  9. Ali, 2023, Performance of ChatGPT and GPT-4 on neurosurgery written board examinations, Neurosurgery, 93, 1353, 10.1227/neu.0000000000002632 · DOI · Paper page
  10. Long, 2024, A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study, JMIR Med Educ, 10, 10.2196/49970 · DOI · Paper page
  11. Beşler, 2024, Evaluating GPT-4o’s performance in the official European Board of Radiology exam: a Comprehensive Assessment, Acad Radiol, 31, 4365, 10.1016/j.acra.2024.09.005 · DOI
  12. Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education in prostate cancer: Multimetric assessment, J Med Internet Res, 26, 10.2196/55939 · DOI · Paper page
  13. Mukherjee, 2024, Evaluation of GPT large language model performance on RSNA 2023 case of the day questions, Radiology, 313, 10.1148/radiol.240609 · DOI
  14. Kim, 2024, ChatGPT vision for radiological interpretation: an investigation using medical school radiology examinations, Korean J Radiol, 25, 403, 10.3348/kjr.2024.0017 · DOI
  15. Khan, 2021, A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability, Lancet Digit Health, 3, e7
  16. Hurley, 2024, Evaluation of high-quality information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery, Arthroscopy, 40, 726, 10.1016/j.arthro.2023.07.048 · DOI · Paper page
  17. Onder, 2024, Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy, Sci Rep, 14, 243, 10.1038/s41598-023-50884-w · DOI
  18. Marshall, 2024, Investigating the accuracy and completeness of an artificial intelligence large language model about uveitis: an evaluation of ChatGPT, Ocul Immunol Inflamm, 32, 2052, 10.1080/09273948.2024.2317417 · DOI
  19. Li W, Li L, Xiang T, et al. Can multiple-choice questions really be useful in detecting the abilities of LLMs? 2024. doi:10.48550/ARXIV.2403.17752. · arXiv · doi:10.48550/arxiv.2403.17752
  20. Cheng, 2023, Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study, J Med Internet Res, 25, 10.2196/51229 · DOI · Paper page
  21. Hake, 2024, Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts, Ann Fam Med, 22, 113, 10.1370/afm.3075 · DOI · Paper page
  22. Caterson, 2024, Application of generative language models to orthopaedic practice, BMJ Open, 14, 10.1136/bmjopen-2023-076484 · DOI
  23. Chung, 2023, Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients, Digital Health, 9, 10.1177/20552076231221620 · DOI · Paper page
  24. Chen, 2023, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics, 39, btad557, 10.1093/bioinformatics/btad557 · DOI · Paper page
  25. Jahan, 2024, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Comput Biol Med, 171, 10.1016/j.compbiomed.2024.108189 · DOI · Paper page
  26. Gu, 2024, Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model, Artif Intell Med, 150, 10.1016/j.artmed.2024.102822 · DOI
  27. Cinquin, 2024, ChIP-GPT: a managed large language model for robust data extraction from biomedical database records, Brief Bioinform, 25, bbad535, 10.1093/bib/bbad535 · DOI
  28. Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
  29. Ge, 2024, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology, 80, 1158, 10.1097/HEP.0000000000000834 · DOI · Paper page
  30. Zhu, 2023, Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?, J Transl Med, 21, 269, 10.1186/s12967-023-04123-5 · DOI · Paper page
  31. Huang, 2024, Assessment of a large language model’s responses to questions and cases about glaucoma and retina management, JAMA Ophthalmol, 142, 393, 10.1001/jamaophthalmol.2023.6917 · DOI · Paper page
  32. Civettini, 2024, Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making, Br J Haematol, 204, 1523, 10.1111/bjh.19200 · DOI
  33. Sorin, 2023, Large language model (ChatGPT) as a support tool for breast tumor board, NPJ Breast Cancer, 9, 44, 10.1038/s41523-023-00557-8 · DOI · Paper page
  34. Ali, 2023, ChatGPT and lacrimal drainage disorders: performance and scope of improvement, Ophthalmic Plast Reconstr Surg, 39, 221, 10.1097/IOP.0000000000002418 · DOI · Paper page
  35. Potapenko, 2023, Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT, Acta Ophthalmol, 101, 829, 10.1111/aos.15661 · DOI · Paper page
  36. Rasmussen, 2023, Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis, Graefes Arch Clin Exp Ophthalmol, 261, 3041, 10.1007/s00417-023-06078-1 · DOI
  37. Wilhelm, 2023, Large Language models for therapy recommendations across 3 clinical specialties: comparative study, J Med Internet Res, 25, 10.2196/49324 · DOI · Paper page
  38. Uz, 2023, Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases?, Int J Rheum Dis, 26, 1343, 10.1111/1756-185X.14749 · DOI · Paper page
  39. Cao, 2023, Accuracy of information provided by ChatGPT regarding liver cancer surveillance and diagnosis, AJR Am J Roentgenol, 221, 556, 10.2214/AJR.23.29493 · DOI · Paper page
  40. Balel, 2023, Can ChatGPT be used in oral and maxillofacial surgery?, J Stomatol Oral Maxillofac Surg, 124, 10.1016/j.jormas.2023.101471 · DOI · Paper page
  41. Munoz-Zuluaga, 2023, Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine, Clin Chem, 69, 939, 10.1093/clinchem/hvad058 · DOI · Paper page
  42. Young, 2023, The utility of ChatGPT in generating patient-facing and clinical responses for melanoma, J Am Acad Dermatol, 89, 602, 10.1016/j.jaad.2023.05.024 · DOI
  43. Cadamuro, 2023, Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results, Clin Chem Lab Med, 61, 1158, 10.1515/cclm-2023-0355 · DOI · Paper page
  44. Wan, 2024, Med-unic: unifying cross-lingual medical vision-language pre-training by diminishing bias, Adv Neural Inf Process Syst, 36
  45. Liu, 2024, A systematic evaluation of GPT-4V’s multimodal capability for chest X-ray image analysis, Meta-Radiology, 2, 10.1016/j.metrad.2024.100099 · DOI
  46. Wang, 2023, R2gengpt: radiology report generation with frozen llms, Meta-Radiology, 1, 10.1016/j.metrad.2023.100033 · DOI · Paper page
  47. Lin, 2023, Medical visual question answering: a survey, Artif Intell Med, 143, 10.1016/j.artmed.2023.102611 · DOI · Paper page
  48. Li, 2023, A comprehensive sstudy of GPT-4V’s multimodal capabilities in medical imaging, medRxiv, 2023
  49. Gu Z, Yin C, Liu F, et al. MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. 2024. doi:10.48550/ARXIV.2407.02730. · DOI
  50. Wu, 2024, Hallucination benchmark in medical visual question answering
  51. Oh, 2025, LLM-driven multimodal target volume contouring in radiation oncology, Nat Commun, 16, 718, 10.1038/s41467-025-55963-2 · doi:10.1038/s41467-025-55963-2
  52. Harrer, 2024, Generative AI agents are transforming biology research: high resolution functional genome annotation for multiscale understanding of life, EBioMedicine, 109, 10.1016/j.ebiom.2024.105446 · DOI
  53. Wang, 2025, A survey of LLM-based agents in medicine: how far are we from Baymax?
  54. Liu, 2024, AgentBench: evaluating LLMs as agents
  55. Zheng, 2024, Can modern LLMs act as agent cores in radiology environments?
  56. Rjoob, 2022, Machine learning and the electrocardiogram over two decades: Time series and meta-analysis of the algorithms, evaluation metrics and applications, Artif Intell Med, 132, 10.1016/j.artmed.2022.102381 · DOI · Paper page
  57. Chen, 2024, ChatFFA: an ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography, iScience, 27, 10.1016/j.isci.2024.110021 · DOI
  58. Zhao, 2024, Slit lamp report generation and question answering: development and validation of a multimodal transformer model with large language model integration, J Med Internet Res, 26, 10.2196/54047 · DOI
  59. Charnock, 1999, DISCERN: an instrument for judging the quality of written consumer health information on treatment choices, J Epidemiol Community Health, 53, 105, 10.1136/jech.53.2.105 · DOI · Paper page
  60. Silberg, 1997, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware, JAMA, 277, 1244, 10.1001/jama.1997.03540390074039 · DOI · Paper page
  61. Ozduran, 2022, Evaluating the readability, quality and reliability of online patient education materials on post-covid pain, PeerJ, 10, 10.7717/peerj.13686 · DOI
  62. Rensis, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 5
  63. Rau, 2023, A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines, Radiology, 308, 10.1148/radiol.230970 · DOI · Paper page
  64. Xu, 2025, DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning
  65. Resnikoff, 2020, Estimated number of ophthalmologists worldwide (International Council of Ophthalmology update): will we meet the needs?, Br J Ophthalmol, 104, 588, 10.1136/bjophthalmol-2019-314336 · DOI · Paper page
  66. Yang SH, Zhao HJ, Zhu GY, et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2024AAAI 2024;38(17):19368-19376. doi: 10.1609/aaai.v38i17.29907. · DOI · Paper page

Cited by

See all 68

Metadata

Authors
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D.
Venue
Intelligent Medicine
Publisher
Elsevier BV
Published
2025-05
Volume
5
Issue
2
Pages
151-163
ISSN
  • 2096-9376 (ISSN-L)
  • 2667-1026
OpenAlex
W4409091111
Cited by
68
References
69
FWCI
13.92
Type
journal article
Language
en
Open access
gold
Licence
Funders
The Hong Kong Polytechnic University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.