CoChat

Evaluating large language models and agents in healthcare: key challenges in clinical applications

Open access
Chen, Xiaolan, Xiang, Jiayang, Lu, Shanfu, Liu, Yexin, He, Mingguang, Shi, Danli · 2025 · Intelligent Medicine
Cited by 68

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have emerged as transformative tools with significant potential across healthcare and medicine. In clinical settings, they hold promises for tasks ranging from clinical decision support to patient education. Advances in LLM agents further broaden their utility by enabling multimodal processing and multitask handling in complex clinical workflows. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the high-risk nature of healthcare and the complexity of medical data. This paper provides a comprehensive overview of current evaluation practices for LLMs and LLM agents in medicine. We contributed 3 main aspects: First, we summarized data sources used in evaluations, including existing medical resources and manually designed clinical questions, offering a basis for LLM evaluation in medical settings. Second, we analyzed key medical task scenarios: closed-ended tasks, open-ended tasks, image processing tasks, and real-world multitask scenarios involving LLM agents, thereby offering guidance for further research across different medical applications. Third, we compared evaluation methods and dimensions, covering both automated metrics and human expert assessments, while addressing traditional accuracy measures alongside agent-specific dimensions, such as tool usage and reasoning capabilities. Finally, we identified key challenges and opportunities in this evolving field, emphasizing the need for continued research and interdisciplinary collaboration between healthcare professionals and computer scientists to ensure safe, ethical, and effective deployment of LLMs in clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.imed.2025.03.002

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 63

  1. Ozkaya, 2023, Application of large language models to software engineering tasks: opportunities, risks, and implications, IEEE Softw, 40, 4, 10.1109/MS.2023.3248401 · DOI · Paper page
  2. Demszky, 2023, Using large language models in psychology, Nat Rev Psychol, 10.1038/s44159-023-00241-5 · DOI · Paper page
  3. Zhang, 2024, Examining the role of large language models in orthopedics: systematic review, J Med Internet Res, 26, 10.2196/59607 · DOI
  4. Chen, 2024, EyeGPT for patient inquiries and medical education: development and validation of an ophthalmology large language model, J Med Internet Res, 26, 10.2196/60063 · DOI
  5. Zhang H, Chen J, Jiang F, et al. HuatuoGPT, towards taming language model to be a doctor. 2023. doi:10.48550/ARXIV.2305.15075. · DOI · Paper page
  6. Mihalache, 2024, Accuracy of an artificial intelligence Chatbot’s interpretation of clinical ophthalmic images, JAMA Ophthalmol, 142, 321, 10.1001/jamaophthalmol.2024.0017 · DOI
  7. Wang, 2023, Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI, Int J Med Inform, 177, 10.1016/j.ijmedinf.2023.105173 · DOI · Paper page
  8. Ali, 2023, Performance of ChatGPT and GPT-4 on neurosurgery written board examinations, Neurosurgery, 93, 1353, 10.1227/neu.0000000000002632 · DOI · Paper page
  9. Long, 2024, A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study, JMIR Med Educ, 10, 10.2196/49970 · DOI · Paper page
  10. Beşler, 2024, Evaluating GPT-4o’s performance in the official European Board of Radiology exam: a Comprehensive Assessment, Acad Radiol, 31, 4365, 10.1016/j.acra.2024.09.005 · DOI
  11. Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education in prostate cancer: Multimetric assessment, J Med Internet Res, 26, 10.2196/55939 · DOI · Paper page
  12. Mukherjee, 2024, Evaluation of GPT large language model performance on RSNA 2023 case of the day questions, Radiology, 313, 10.1148/radiol.240609 · DOI
  13. Kim, 2024, ChatGPT vision for radiological interpretation: an investigation using medical school radiology examinations, Korean J Radiol, 25, 403, 10.3348/kjr.2024.0017 · DOI
  14. Khan, 2021, A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability, Lancet Digit Health, 3, e7
  15. Hurley, 2024, Evaluation of high-quality information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery, Arthroscopy, 40, 726, 10.1016/j.arthro.2023.07.048 · DOI · Paper page
  16. Onder, 2024, Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy, Sci Rep, 14, 243, 10.1038/s41598-023-50884-w · DOI
  17. Marshall, 2024, Investigating the accuracy and completeness of an artificial intelligence large language model about uveitis: an evaluation of ChatGPT, Ocul Immunol Inflamm, 32, 2052, 10.1080/09273948.2024.2317417 · DOI
  18. Li W, Li L, Xiang T, et al. Can multiple-choice questions really be useful in detecting the abilities of LLMs? 2024. doi:10.48550/ARXIV.2403.17752. · arXiv · doi:10.48550/arxiv.2403.17752
  19. Cheng, 2023, Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study, J Med Internet Res, 25, 10.2196/51229 · DOI · Paper page
  20. Hake, 2024, Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts, Ann Fam Med, 22, 113, 10.1370/afm.3075 · DOI · Paper page
  21. Caterson, 2024, Application of generative language models to orthopaedic practice, BMJ Open, 14, 10.1136/bmjopen-2023-076484 · DOI
  22. Chung, 2023, Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients, Digital Health, 9, 10.1177/20552076231221620 · DOI · Paper page
  23. Chen, 2023, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics, 39, btad557, 10.1093/bioinformatics/btad557 · DOI · Paper page
  24. Jahan, 2024, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Comput Biol Med, 171, 10.1016/j.compbiomed.2024.108189 · DOI · Paper page
  25. Gu, 2024, Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model, Artif Intell Med, 150, 10.1016/j.artmed.2024.102822 · DOI
  26. Cinquin, 2024, ChIP-GPT: a managed large language model for robust data extraction from biomedical database records, Brief Bioinform, 25, bbad535, 10.1093/bib/bbad535 · DOI
  27. Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
  28. Ge, 2024, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology, 80, 1158, 10.1097/HEP.0000000000000834 · DOI · Paper page
  29. Zhu, 2023, Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?, J Transl Med, 21, 269, 10.1186/s12967-023-04123-5 · DOI · Paper page
  30. Huang, 2024, Assessment of a large language model’s responses to questions and cases about glaucoma and retina management, JAMA Ophthalmol, 142, 393, 10.1001/jamaophthalmol.2023.6917 · DOI · Paper page
  31. Civettini, 2024, Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making, Br J Haematol, 204, 1523, 10.1111/bjh.19200 · DOI
  32. Sorin, 2023, Large language model (ChatGPT) as a support tool for breast tumor board, NPJ Breast Cancer, 9, 44, 10.1038/s41523-023-00557-8 · DOI · Paper page
  33. Ali, 2023, ChatGPT and lacrimal drainage disorders: performance and scope of improvement, Ophthalmic Plast Reconstr Surg, 39, 221, 10.1097/IOP.0000000000002418 · DOI · Paper page
  34. Potapenko, 2023, Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT, Acta Ophthalmol, 101, 829, 10.1111/aos.15661 · DOI · Paper page
  35. Rasmussen, 2023, Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis, Graefes Arch Clin Exp Ophthalmol, 261, 3041, 10.1007/s00417-023-06078-1 · DOI
  36. Wilhelm, 2023, Large Language models for therapy recommendations across 3 clinical specialties: comparative study, J Med Internet Res, 25, 10.2196/49324 · DOI · Paper page
  37. Uz, 2023, Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases?, Int J Rheum Dis, 26, 1343, 10.1111/1756-185X.14749 · DOI · Paper page
  38. Cao, 2023, Accuracy of information provided by ChatGPT regarding liver cancer surveillance and diagnosis, AJR Am J Roentgenol, 221, 556, 10.2214/AJR.23.29493 · DOI · Paper page
  39. Balel, 2023, Can ChatGPT be used in oral and maxillofacial surgery?, J Stomatol Oral Maxillofac Surg, 124, 10.1016/j.jormas.2023.101471 · DOI · Paper page
  40. Munoz-Zuluaga, 2023, Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine, Clin Chem, 69, 939, 10.1093/clinchem/hvad058 · DOI · Paper page
  41. Young, 2023, The utility of ChatGPT in generating patient-facing and clinical responses for melanoma, J Am Acad Dermatol, 89, 602, 10.1016/j.jaad.2023.05.024 · DOI
  42. Cadamuro, 2023, Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results, Clin Chem Lab Med, 61, 1158, 10.1515/cclm-2023-0355 · DOI · Paper page
  43. Wan, 2024, Med-unic: unifying cross-lingual medical vision-language pre-training by diminishing bias, Adv Neural Inf Process Syst, 36
  44. Liu, 2024, A systematic evaluation of GPT-4V’s multimodal capability for chest X-ray image analysis, Meta-Radiology, 2, 10.1016/j.metrad.2024.100099 · DOI
  45. Wang, 2023, R2gengpt: radiology report generation with frozen llms, Meta-Radiology, 1, 10.1016/j.metrad.2023.100033 · DOI · Paper page
  46. Lin, 2023, Medical visual question answering: a survey, Artif Intell Med, 143, 10.1016/j.artmed.2023.102611 · DOI · Paper page
  47. Li, 2023, A comprehensive sstudy of GPT-4V’s multimodal capabilities in medical imaging, medRxiv, 2023
  48. Gu Z, Yin C, Liu F, et al. MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. 2024. doi:10.48550/ARXIV.2407.02730. · DOI
  49. Wu, 2024, Hallucination benchmark in medical visual question answering
  50. Oh, 2025, LLM-driven multimodal target volume contouring in radiation oncology, Nat Commun, 16, 718, 10.1038/s41467-025-55963-2 · doi:10.1038/s41467-025-55963-2
  51. Harrer, 2024, Generative AI agents are transforming biology research: high resolution functional genome annotation for multiscale understanding of life, EBioMedicine, 109, 10.1016/j.ebiom.2024.105446 · DOI
  52. Wang, 2025, A survey of LLM-based agents in medicine: how far are we from Baymax?
  53. Liu, 2024, AgentBench: evaluating LLMs as agents
  54. Zheng, 2024, Can modern LLMs act as agent cores in radiology environments?
  55. Rjoob, 2022, Machine learning and the electrocardiogram over two decades: Time series and meta-analysis of the algorithms, evaluation metrics and applications, Artif Intell Med, 132, 10.1016/j.artmed.2022.102381 · DOI · Paper page
  56. Chen, 2024, ChatFFA: an ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography, iScience, 27, 10.1016/j.isci.2024.110021 · DOI
  57. Zhao, 2024, Slit lamp report generation and question answering: development and validation of a multimodal transformer model with large language model integration, J Med Internet Res, 26, 10.2196/54047 · DOI
  58. Silberg, 1997, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware, JAMA, 277, 1244, 10.1001/jama.1997.03540390074039 · DOI · Paper page
  59. Ozduran, 2022, Evaluating the readability, quality and reliability of online patient education materials on post-covid pain, PeerJ, 10, 10.7717/peerj.13686 · DOI
  60. Rensis, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 5
  61. Rau, 2023, A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines, Radiology, 308, 10.1148/radiol.230970 · DOI · Paper page
  62. Xu, 2025, DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning
  63. Yang SH, Zhao HJ, Zhu GY, et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2024AAAI 2024;38(17):19368-19376. doi: 10.1609/aaai.v38i17.29907. · DOI · Paper page

Cited by

Metadata

Authors
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D.
Venue
Intelligent Medicine
Publisher
Elsevier BV
Published
2025-05
Volume
5
Issue
2
Pages
151-163
ISSN
  • 2096-9376 (ISSN-L)
  • 2667-1026
OpenAlex
W4409091111
Cited by
68
References
74
FWCI
13.92
Type
journal article
Language
en
Open access
gold
Licence
Funders
The Hong Kong Polytechnic University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.