CoChat

Evaluating large language models and agents in healthcare: key challenges in clinical applications

Open access
Chen, Xiaolan, Xiang, Jiayang, Lu, Shanfu, Liu, Yexin, He, Mingguang, Shi, Danli · 2025 · Intelligent Medicine
Cited by 68

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have emerged as transformative tools with significant potential across healthcare and medicine. In clinical settings, they hold promises for tasks ranging from clinical decision support to patient education. Advances in LLM agents further broaden their utility by enabling multimodal processing and multitask handling in complex clinical workflows. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the high-risk nature of healthcare and the complexity of medical data. This paper provides a comprehensive overview of current evaluation practices for LLMs and LLM agents in medicine. We contributed 3 main aspects: First, we summarized data sources used in evaluations, including existing medical resources and manually designed clinical questions, offering a basis for LLM evaluation in medical settings. Second, we analyzed key medical task scenarios: closed-ended tasks, open-ended tasks, image processing tasks, and real-world multitask scenarios involving LLM agents, thereby offering guidance for further research across different medical applications. Third, we compared evaluation methods and dimensions, covering both automated metrics and human expert assessments, while addressing traditional accuracy measures alongside agent-specific dimensions, such as tool usage and reasoning capabilities. Finally, we identified key challenges and opportunities in this evolving field, emphasizing the need for continued research and interdisciplinary collaboration between healthcare professionals and computer scientists to ensure safe, ethical, and effective deployment of LLMs in clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.imed.2025.03.002

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 62 references

Research works printed but not linked: 70

  1. Ozkaya, 2023, Application of large language models to software engineering tasks: opportunities, risks, and implications, IEEE Softw, 40, 4, 10.1109/MS.2023.3248401 · DOI · Paper page
  2. Alqahtani, 2023, The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research, Res Social Adm Pharm, 19, 1236, 10.1016/j.sapharm.2023.05.016 · DOI · Paper page
  3. Thirunavukarasu, 2023, Large language models in medicine, Nat Med, 29, 1930, 10.1038/s41591-023-02448-8 · DOI · Paper page
  4. Demszky, 2023, Using large language models in psychology, Nat Rev Psychol, 10.1038/s44159-023-00241-5 · DOI · Paper page
  5. Zhang, 2024, Examining the role of large language models in orthopedics: systematic review, J Med Internet Res, 26, 10.2196/59607 · DOI
  6. Chen, 2024, EyeGPT for patient inquiries and medical education: development and validation of an ophthalmology large language model, J Med Internet Res, 26, 10.2196/60063 · DOI
  7. Zhang H, Chen J, Jiang F, et al. HuatuoGPT, towards taming language model to be a doctor. 2023. doi:10.48550/ARXIV.2305.15075. · DOI · Paper page
  8. Mihalache, 2024, Accuracy of an artificial intelligence Chatbot’s interpretation of clinical ophthalmic images, JAMA Ophthalmol, 142, 321, 10.1001/jamaophthalmol.2024.0017 · DOI
  9. Wang, 2023, Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI, Int J Med Inform, 177, 10.1016/j.ijmedinf.2023.105173 · DOI
  10. Antaki, 2023, Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings, Ophthalmol Sci, 3, 10.1016/j.xops.2023.100324 · DOI · Paper page
  11. Ali, 2023, Performance of ChatGPT and GPT-4 on neurosurgery written board examinations, Neurosurgery, 93, 1353, 10.1227/neu.0000000000002632 · DOI · Paper page
  12. Long, 2024, A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study, JMIR Med Educ, 10, 10.2196/49970 · DOI · Paper page
  13. Beşler, 2024, Evaluating GPT-4o’s performance in the official European Board of Radiology exam: a Comprehensive Assessment, Acad Radiol, 31, 4365, 10.1016/j.acra.2024.09.005 · DOI
  14. Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education in prostate cancer: Multimetric assessment, J Med Internet Res, 26, 10.2196/55939 · DOI · Paper page
  15. Mukherjee, 2024, Evaluation of GPT large language model performance on RSNA 2023 case of the day questions, Radiology, 313, 10.1148/radiol.240609 · DOI
  16. Kim, 2024, ChatGPT vision for radiological interpretation: an investigation using medical school radiology examinations, Korean J Radiol, 25, 403, 10.3348/kjr.2024.0017 · DOI
  17. Khan, 2021, A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability, Lancet Digit Health, 3, e7
  18. Hurley, 2024, Evaluation of high-quality information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery, Arthroscopy, 40, 726, 10.1016/j.arthro.2023.07.048 · DOI · Paper page
  19. Onder, 2024, Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy, Sci Rep, 14, 243, 10.1038/s41598-023-50884-w · DOI
  20. Marshall, 2024, Investigating the accuracy and completeness of an artificial intelligence large language model about uveitis: an evaluation of ChatGPT, Ocul Immunol Inflamm, 32, 2052, 10.1080/09273948.2024.2317417 · DOI
  21. Li W, Li L, Xiang T, et al. Can multiple-choice questions really be useful in detecting the abilities of LLMs? 2024. doi:10.48550/ARXIV.2403.17752. · arXiv · doi:10.48550/arxiv.2403.17752
  22. Cheng, 2023, Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study, J Med Internet Res, 25, 10.2196/51229 · DOI · Paper page
  23. Hake, 2024, Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts, Ann Fam Med, 22, 113, 10.1370/afm.3075 · DOI · Paper page
  24. Caterson, 2024, Application of generative language models to orthopaedic practice, BMJ Open, 14, 10.1136/bmjopen-2023-076484 · DOI
  25. Chung, 2023, Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients, Digital Health, 9, 10.1177/20552076231221620 · DOI · Paper page
  26. Chen, 2023, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics, 39, btad557, 10.1093/bioinformatics/btad557 · DOI · Paper page
  27. Jahan, 2024, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Comput Biol Med, 171, 10.1016/j.compbiomed.2024.108189 · DOI · Paper page
  28. Gu, 2024, Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model, Artif Intell Med, 150, 10.1016/j.artmed.2024.102822 · DOI
  29. Cinquin, 2024, ChIP-GPT: a managed large language model for robust data extraction from biomedical database records, Brief Bioinform, 25, bbad535, 10.1093/bib/bbad535 · DOI
  30. Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
  31. Ge, 2024, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology, 80, 1158, 10.1097/HEP.0000000000000834 · DOI · Paper page
  32. Zhu, 2023, Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?, J Transl Med, 21, 269, 10.1186/s12967-023-04123-5 · DOI
  33. Huang, 2024, Assessment of a large language model’s responses to questions and cases about glaucoma and retina management, JAMA Ophthalmol, 142, 393, 10.1001/jamaophthalmol.2023.6917 · DOI · Paper page
  34. Civettini, 2024, Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making, Br J Haematol, 204, 1523, 10.1111/bjh.19200 · DOI
  35. Sorin, 2023, Large language model (ChatGPT) as a support tool for breast tumor board, NPJ Breast Cancer, 9, 44, 10.1038/s41523-023-00557-8 · DOI · Paper page
  36. Ali, 2023, ChatGPT and lacrimal drainage disorders: performance and scope of improvement, Ophthalmic Plast Reconstr Surg, 39, 221, 10.1097/IOP.0000000000002418 · DOI · Paper page
  37. Potapenko, 2023, Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT, Acta Ophthalmol, 101, 829, 10.1111/aos.15661 · DOI
  38. Rasmussen, 2023, Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis, Graefes Arch Clin Exp Ophthalmol, 261, 3041, 10.1007/s00417-023-06078-1 · DOI
  39. Wilhelm, 2023, Large Language models for therapy recommendations across 3 clinical specialties: comparative study, J Med Internet Res, 25, 10.2196/49324 · DOI · Paper page
  40. Uz, 2023, Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases?, Int J Rheum Dis, 26, 1343, 10.1111/1756-185X.14749 · DOI
  41. Cao, 2023, Accuracy of information provided by ChatGPT regarding liver cancer surveillance and diagnosis, AJR Am J Roentgenol, 221, 556, 10.2214/AJR.23.29493 · DOI · Paper page
  42. Balel, 2023, Can ChatGPT be used in oral and maxillofacial surgery?, J Stomatol Oral Maxillofac Surg, 124, 10.1016/j.jormas.2023.101471 · DOI · Paper page
  43. Munoz-Zuluaga, 2023, Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine, Clin Chem, 69, 939, 10.1093/clinchem/hvad058 · DOI · Paper page
  44. Young, 2023, The utility of ChatGPT in generating patient-facing and clinical responses for melanoma, J Am Acad Dermatol, 89, 602, 10.1016/j.jaad.2023.05.024 · DOI
  45. Cadamuro, 2023, Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results, Clin Chem Lab Med, 61, 1158, 10.1515/cclm-2023-0355 · DOI · Paper page
  46. Wan, 2024, Med-unic: unifying cross-lingual medical vision-language pre-training by diminishing bias, Adv Neural Inf Process Syst, 36
  47. Liu, 2024, A systematic evaluation of GPT-4V’s multimodal capability for chest X-ray image analysis, Meta-Radiology, 2, 10.1016/j.metrad.2024.100099 · DOI
  48. Wang, 2023, R2gengpt: radiology report generation with frozen llms, Meta-Radiology, 1, 10.1016/j.metrad.2023.100033 · DOI · Paper page
  49. Lin, 2023, Medical visual question answering: a survey, Artif Intell Med, 143, 10.1016/j.artmed.2023.102611 · DOI · Paper page
  50. Li, 2023, A comprehensive sstudy of GPT-4V’s multimodal capabilities in medical imaging, medRxiv, 2023
  51. Gu Z, Yin C, Liu F, et al. MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. 2024. doi:10.48550/ARXIV.2407.02730. · DOI
  52. Wu, 2024, Hallucination benchmark in medical visual question answering
  53. Oh, 2025, LLM-driven multimodal target volume contouring in radiation oncology, Nat Commun, 16, 718, 10.1038/s41467-025-55963-2 · doi:10.1038/s41467-025-55963-2
  54. Harrer, 2024, Generative AI agents are transforming biology research: high resolution functional genome annotation for multiscale understanding of life, EBioMedicine, 109, 10.1016/j.ebiom.2024.105446 · DOI
  55. Wang, 2025, A survey of LLM-based agents in medicine: how far are we from Baymax?
  56. Liu, 2024, AgentBench: evaluating LLMs as agents
  57. Zheng, 2024, Can modern LLMs act as agent cores in radiology environments?
  58. Rjoob, 2022, Machine learning and the electrocardiogram over two decades: Time series and meta-analysis of the algorithms, evaluation metrics and applications, Artif Intell Med, 132, 10.1016/j.artmed.2022.102381 · DOI
  59. Chen, 2024, ChatFFA: an ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography, iScience, 27, 10.1016/j.isci.2024.110021 · DOI
  60. Zhao, 2024, Slit lamp report generation and question answering: development and validation of a multimodal transformer model with large language model integration, J Med Internet Res, 26, 10.2196/54047 · DOI
  61. Charnock, 1999, DISCERN: an instrument for judging the quality of written consumer health information on treatment choices, J Epidemiol Community Health, 53, 105, 10.1136/jech.53.2.105 · DOI · Paper page
  62. Silberg, 1997, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware, JAMA, 277, 1244, 10.1001/jama.1997.03540390074039 · DOI · Paper page
  63. Ozduran, 2022, Evaluating the readability, quality and reliability of online patient education materials on post-covid pain, PeerJ, 10, 10.7717/peerj.13686 · DOI
  64. Rensis, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 5
  65. Wang, 2024, Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: comparative study, J Med Internet Res, 26, 10.2196/58041 · DOI · Paper page
  66. Rau, 2023, A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines, Radiology, 308, 10.1148/radiol.230970 · DOI · Paper page
  67. Xu, 2025, DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning
  68. Resnikoff, 2020, Estimated number of ophthalmologists worldwide (International Council of Ophthalmology update): will we meet the needs?, Br J Ophthalmol, 104, 588, 10.1136/bjophthalmol-2019-314336 · DOI · Paper page
  69. Yang SH, Zhao HJ, Zhu GY, et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2024AAAI 2024;38(17):19368-19376. doi: 10.1609/aaai.v38i17.29907. · DOI · Paper page
  70. Li, 2024, Integrated image-based deep learning and language models for primary diabetes care, Nat Med, 30, 2886, 10.1038/s41591-024-03139-8 · DOI · Paper page

Cited by

See all 68

Metadata

Authors
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D.
Venue
Intelligent Medicine
Publisher
Elsevier BV
Published
2025-05
Volume
5
Issue
2
Pages
151-163
ISSN
  • 2096-9376 (ISSN-L)
  • 2667-1026
OpenAlex
W4409091111
Cited by
68
References
62
FWCI
13.92
Type
journal article
Language
en
Open access
gold
Licence
Funders
The Hong Kong Polytechnic University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluating large language models and agents in healthcare: key challenges in clinical applications. https://app.cochat.ai/papers/10.1016/j.imed.2025.03.002
The paper
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025). Evaluating large language models and agents in healthcare: key challenges in clinical applications. Intelligent Medicine. https://doi.org/10.1016/j.imed.2025.03.002

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.