Evaluating large language models and agents in healthcare: key challenges in clinical applications
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have emerged as transformative tools with significant potential across healthcare and medicine. In clinical settings, they hold promises for tasks ranging from clinical decision support to patient education. Advances in LLM agents further broaden their utility by enabling multimodal processing and multitask handling in complex clinical workflows. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the high-risk nature of healthcare and the complexity of medical data. This paper provides a comprehensive overview of current evaluation practices for LLMs and LLM agents in medicine. We contributed 3 main aspects: First, we summarized data sources used in evaluations, including existing medical resources and manually designed clinical questions, offering a basis for LLM evaluation in medical settings. Second, we analyzed key medical task scenarios: closed-ended tasks, open-ended tasks, image processing tasks, and real-world multitask scenarios involving LLM agents, thereby offering guidance for further research across different medical applications. Third, we compared evaluation methods and dimensions, covering both automated metrics and human expert assessments, while addressing traditional accuracy measures alongside agent-specific dimensions, such as tool usage and reasoning capabilities. Finally, we identified key challenges and opportunities in this evolving field, emphasizing the need for continued research and interdisciplinary collaboration between healthcare professionals and computer scientists to ensure safe, ethical, and effective deployment of LLMs in clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citationLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
The Medical Segmentation Decathlon
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30695-9 · Cited by 1,384
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738
Self-citationChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662
Self-citationEvaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483
Self-citationTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
Self-citationBhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424
Evaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423
Evaluating Large Language Models on Medical Evidence Summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J., et al. (2023) · doi:10.1101/2023.04.22.23288967 · Cited by 423
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · arXiv · doi:10.48550/arxiv.2303.09038 · Cited by 393
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-43436-9 · Cited by 319
Empowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315
Zaretsky, J., Kim, J. M., Baskharoun, S., Zhao, Y., Austrian, J., Aphinyanaphongs, Y., Gupta, R., Blecker, S. B., & Feldman, J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.0357 · Cited by 303
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282
Large language models to identify social determinants of health in electronic health records
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00970-0 · Cited by 278
Integrated image-based deep learning and language models for primary diabetes care
Li, J., Guan, Z., Wang, J., Cheung, C. Y., Zheng, Y., Lim, L.-L., Lim, C. C., Ruamviboonsuk, P., Raman, R., Corsino, L., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03139-8 · Cited by 250
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
ChatGPT and antimicrobial advice: the end of the consulting infection doctor?
Howard, A., Hope, W., & Gerada, A. (2023) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(23)00113-5 · Cited by 202
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201
Xie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198
Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
Holmes, J., Liu, Z., Zhang, L., Ding, Y., Sio, T. T., McGee, L. A., Ashman, J. B., Li, X., Liu, T., Shen, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01938 · Cited by 190
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187
Decker, H., Trang, K., Ramirez, J., Colley, A., Pierce, L., Coleman, M., Bongiovanni, T., Melton, G. B., & Wick, E. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36997 · Cited by 178
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 172
Analysis of large-language model versus human performance for genetics questions
Duong, D., & Solomon, B. D. (2024) · European Journal of Human Genetics · doi:10.1038/s41431-023-01396-8 · Cited by 159
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146
Evaluating large language models as agents in the clinic
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01083-y · Cited by 143
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139
Performance of Generative Large Language Models on Ophthalmology Board–Style Questions
Cai, L. Z., Shaheen, A., Jin, A., Fukui, R., Yi, J. S., Yannuzzi, N., & Alabiad, C. (2023) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2023.05.024 · Cited by 139
Menz, B. D., Kuderer, N. M., Bacchi, S., Modi, N. D., Chin-Yee, B., Hu, T., Rickard, C., Haseloff, M., Vitry, A., McKinnon, R. A., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078538 · Cited by 132
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
Zong, H., Li, J., Wu, E., Wu, R., Lu, J., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05125-7 · Cited by 114
Evaluating the use of large language model in identifying top research questions in gastroenterology
Lahat, A., Shachar, E., Avidan, B., Shatz, Z., Glicksberg, B. S., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-31412-2 · Cited by 111
Performance of ChatGPT on the Plastic Surgery Inservice Training Examination
Gupta, R., Herzog, I., Park, J. B., Weisberger, J., Firouzbakht, P., Ocon, V., Chao, J., Lee, E. S., & Mailey, B. A. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad128 · Cited by 109
Wan, P., Huang, Z., Tang, W., Nie, Y., Pei, D., Deng, S., Chen, J., Zhou, Y., Duan, H., Chen, Q., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03148-7 · Cited by 97
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87
Wang, D., Liang, J., Ye, J., Li, J., Li, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/58041 · Cited by 76
Self-citationFFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72
Self-citationYe, C., Zweck, E., Ma, Z., Smith, J., & Katz, S. (2024) · Arthritis & Rheumatology · doi:10.1002/art.42737 · Cited by 69
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.02483 · Cited by 65
Xu, P., Chen, X., Zhao, Z., & Shi, D. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325054 · Cited by 63
Self-citationMMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54
Dubinski, D., Won, S.-Y., Trnovec, S., Behmanesh, B., Baumgarten, P., Dinc, N., Konczalla, J., Chan, A., Bernstock, J. D., Freiman, T. M., et al. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-05908-3 · Cited by 45
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42
The Evaluation of Generative AI Should Include Repetition to Assess Stability
Zhu, L., Mou, W., Hong, C., Yang, T., Lai, Y., Qi, C., Lin, A., Zhang, J., & Luo, P. (2024) · JMIR mHealth and uHealth · doi:10.2196/57978 · Cited by 38
ICGA-GPT: report generation and question answering for indocyanine green angiography images
Chen, X., Zhang, W., Zhao, Z., Xu, P., Zheng, Y., Shi, D., & He, M. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324446 · Cited by 36
Self-citationBenchmarking Large Language Models on CMExam - A Comprehensive Chinese Medical Exam Dataset
Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., Wang, H., You, C., Guo, Z., Zhu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.03030 · Cited by 36
Performance of ChatGPT on Chinese master’s degree entrance examination in clinical medicine
Li, K.-C., Bu, Z.-J., Shahjalal, M., He, B.-X., Zhuang, Z.-F., Li, C., Liu, J.-P., Wang, B., & Liu, Z.-L. (2024) · PLOS ONE · doi:10.1371/journal.pone.0301702 · Cited by 23
MedRedQA for Medical Consumer Question Answering: Dataset, Tasks, and Neural Baselines
Nguyen, V., Karimi, S., Rybinski, M., & Xing, Z. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.42 · Cited by 5
Research works printed but not linked: 66
- Ozkaya, 2023, Application of large language models to software engineering tasks: opportunities, risks, and implications, IEEE Softw, 40, 4, 10.1109/MS.2023.3248401 · DOI · Paper page
- Alqahtani, 2023, The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research, Res Social Adm Pharm, 19, 1236, 10.1016/j.sapharm.2023.05.016 · DOI · Paper page
- Demszky, 2023, Using large language models in psychology, Nat Rev Psychol, 10.1038/s44159-023-00241-5 · DOI · Paper page
- Zhang, 2024, Examining the role of large language models in orthopedics: systematic review, J Med Internet Res, 26, 10.2196/59607 · DOI
- Chen, 2024, EyeGPT for patient inquiries and medical education: development and validation of an ophthalmology large language model, J Med Internet Res, 26, 10.2196/60063 · DOI
- Zhang H, Chen J, Jiang F, et al. HuatuoGPT, towards taming language model to be a doctor. 2023. doi:10.48550/ARXIV.2305.15075. · DOI · Paper page
- Mihalache, 2024, Accuracy of an artificial intelligence Chatbot’s interpretation of clinical ophthalmic images, JAMA Ophthalmol, 142, 321, 10.1001/jamaophthalmol.2024.0017 · DOI
- Wang, 2023, Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI, Int J Med Inform, 177, 10.1016/j.ijmedinf.2023.105173 · DOI · Paper page
- Ali, 2023, Performance of ChatGPT and GPT-4 on neurosurgery written board examinations, Neurosurgery, 93, 1353, 10.1227/neu.0000000000002632 · DOI · Paper page
- Long, 2024, A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study, JMIR Med Educ, 10, 10.2196/49970 · DOI · Paper page
- Beşler, 2024, Evaluating GPT-4o’s performance in the official European Board of Radiology exam: a Comprehensive Assessment, Acad Radiol, 31, 4365, 10.1016/j.acra.2024.09.005 · DOI
- Gibson, 2024, Evaluating the efficacy of ChatGPT as a patient education in prostate cancer: Multimetric assessment, J Med Internet Res, 26, 10.2196/55939 · DOI · Paper page
- Mukherjee, 2024, Evaluation of GPT large language model performance on RSNA 2023 case of the day questions, Radiology, 313, 10.1148/radiol.240609 · DOI
- Kim, 2024, ChatGPT vision for radiological interpretation: an investigation using medical school radiology examinations, Korean J Radiol, 25, 403, 10.3348/kjr.2024.0017 · DOI
- Khan, 2021, A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability, Lancet Digit Health, 3, e7
- Hurley, 2024, Evaluation of high-quality information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery, Arthroscopy, 40, 726, 10.1016/j.arthro.2023.07.048 · DOI · Paper page
- Onder, 2024, Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy, Sci Rep, 14, 243, 10.1038/s41598-023-50884-w · DOI
- Marshall, 2024, Investigating the accuracy and completeness of an artificial intelligence large language model about uveitis: an evaluation of ChatGPT, Ocul Immunol Inflamm, 32, 2052, 10.1080/09273948.2024.2317417 · DOI
- Li W, Li L, Xiang T, et al. Can multiple-choice questions really be useful in detecting the abilities of LLMs? 2024. doi:10.48550/ARXIV.2403.17752. · arXiv · doi:10.48550/arxiv.2403.17752
- Cheng, 2023, Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study, J Med Internet Res, 25, 10.2196/51229 · DOI · Paper page
- Hake, 2024, Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts, Ann Fam Med, 22, 113, 10.1370/afm.3075 · DOI · Paper page
- Caterson, 2024, Application of generative language models to orthopaedic practice, BMJ Open, 14, 10.1136/bmjopen-2023-076484 · DOI
- Chung, 2023, Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients, Digital Health, 9, 10.1177/20552076231221620 · DOI · Paper page
- Chen, 2023, An extensive benchmark study on biomedical text generation and mining with ChatGPT, Bioinformatics, 39, btad557, 10.1093/bioinformatics/btad557 · DOI · Paper page
- Jahan, 2024, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Comput Biol Med, 171, 10.1016/j.compbiomed.2024.108189 · DOI · Paper page
- Gu, 2024, Automatic quantitative stroke severity assessment based on Chinese clinical named entity recognition with domain-adaptive pre-trained large language model, Artif Intell Med, 150, 10.1016/j.artmed.2024.102822 · DOI
- Cinquin, 2024, ChIP-GPT: a managed large language model for robust data extraction from biomedical database records, Brief Bioinform, 25, bbad535, 10.1093/bib/bbad535 · DOI
- Sarraju, 2023, Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model, JAMA, 329, 842, 10.1001/jama.2023.1044 · DOI · Paper page
- Ge, 2024, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology, 80, 1158, 10.1097/HEP.0000000000000834 · DOI · Paper page
- Zhu, 2023, Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge?, J Transl Med, 21, 269, 10.1186/s12967-023-04123-5 · DOI · Paper page
- Huang, 2024, Assessment of a large language model’s responses to questions and cases about glaucoma and retina management, JAMA Ophthalmol, 142, 393, 10.1001/jamaophthalmol.2023.6917 · DOI · Paper page
- Civettini, 2024, Evaluating the performance of large language models in haematopoietic stem cell transplantation decision-making, Br J Haematol, 204, 1523, 10.1111/bjh.19200 · DOI
- Sorin, 2023, Large language model (ChatGPT) as a support tool for breast tumor board, NPJ Breast Cancer, 9, 44, 10.1038/s41523-023-00557-8 · DOI · Paper page
- Ali, 2023, ChatGPT and lacrimal drainage disorders: performance and scope of improvement, Ophthalmic Plast Reconstr Surg, 39, 221, 10.1097/IOP.0000000000002418 · DOI · Paper page
- Potapenko, 2023, Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT, Acta Ophthalmol, 101, 829, 10.1111/aos.15661 · DOI · Paper page
- Rasmussen, 2023, Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitis, Graefes Arch Clin Exp Ophthalmol, 261, 3041, 10.1007/s00417-023-06078-1 · DOI
- Wilhelm, 2023, Large Language models for therapy recommendations across 3 clinical specialties: comparative study, J Med Internet Res, 25, 10.2196/49324 · DOI · Paper page
- Uz, 2023, Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases?, Int J Rheum Dis, 26, 1343, 10.1111/1756-185X.14749 · DOI · Paper page
- Cao, 2023, Accuracy of information provided by ChatGPT regarding liver cancer surveillance and diagnosis, AJR Am J Roentgenol, 221, 556, 10.2214/AJR.23.29493 · DOI · Paper page
- Balel, 2023, Can ChatGPT be used in oral and maxillofacial surgery?, J Stomatol Oral Maxillofac Surg, 124, 10.1016/j.jormas.2023.101471 · DOI · Paper page
- Munoz-Zuluaga, 2023, Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine, Clin Chem, 69, 939, 10.1093/clinchem/hvad058 · DOI · Paper page
- Young, 2023, The utility of ChatGPT in generating patient-facing and clinical responses for melanoma, J Am Acad Dermatol, 89, 602, 10.1016/j.jaad.2023.05.024 · DOI
- Cadamuro, 2023, Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results, Clin Chem Lab Med, 61, 1158, 10.1515/cclm-2023-0355 · DOI · Paper page
- Wan, 2024, Med-unic: unifying cross-lingual medical vision-language pre-training by diminishing bias, Adv Neural Inf Process Syst, 36
- Liu, 2024, A systematic evaluation of GPT-4V’s multimodal capability for chest X-ray image analysis, Meta-Radiology, 2, 10.1016/j.metrad.2024.100099 · DOI
- Wang, 2023, R2gengpt: radiology report generation with frozen llms, Meta-Radiology, 1, 10.1016/j.metrad.2023.100033 · DOI · Paper page
- Lin, 2023, Medical visual question answering: a survey, Artif Intell Med, 143, 10.1016/j.artmed.2023.102611 · DOI · Paper page
- Li, 2023, A comprehensive sstudy of GPT-4V’s multimodal capabilities in medical imaging, medRxiv, 2023
- Gu Z, Yin C, Liu F, et al. MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. 2024. doi:10.48550/ARXIV.2407.02730. · DOI
- Wu, 2024, Hallucination benchmark in medical visual question answering
- Oh, 2025, LLM-driven multimodal target volume contouring in radiation oncology, Nat Commun, 16, 718, 10.1038/s41467-025-55963-2 · doi:10.1038/s41467-025-55963-2
- Harrer, 2024, Generative AI agents are transforming biology research: high resolution functional genome annotation for multiscale understanding of life, EBioMedicine, 109, 10.1016/j.ebiom.2024.105446 · DOI
- Wang, 2025, A survey of LLM-based agents in medicine: how far are we from Baymax?
- Liu, 2024, AgentBench: evaluating LLMs as agents
- Zheng, 2024, Can modern LLMs act as agent cores in radiology environments?
- Rjoob, 2022, Machine learning and the electrocardiogram over two decades: Time series and meta-analysis of the algorithms, evaluation metrics and applications, Artif Intell Med, 132, 10.1016/j.artmed.2022.102381 · DOI · Paper page
- Chen, 2024, ChatFFA: an ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography, iScience, 27, 10.1016/j.isci.2024.110021 · DOI
- Zhao, 2024, Slit lamp report generation and question answering: development and validation of a multimodal transformer model with large language model integration, J Med Internet Res, 26, 10.2196/54047 · DOI
- Charnock, 1999, DISCERN: an instrument for judging the quality of written consumer health information on treatment choices, J Epidemiol Community Health, 53, 105, 10.1136/jech.53.2.105 · DOI · Paper page
- Silberg, 1997, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware, JAMA, 277, 1244, 10.1001/jama.1997.03540390074039 · DOI · Paper page
- Ozduran, 2022, Evaluating the readability, quality and reliability of online patient education materials on post-covid pain, PeerJ, 10, 10.7717/peerj.13686 · DOI
- Rensis, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 5
- Rau, 2023, A context-based chatbot surpasses trained radiologists and generic ChatGPT in following the ACR appropriateness guidelines, Radiology, 308, 10.1148/radiol.230970 · DOI · Paper page
- Xu, 2025, DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in bilingual complex ophthalmology reasoning
- Resnikoff, 2020, Estimated number of ophthalmologists worldwide (International Council of Ophthalmology update): will we meet the needs?, Br J Ophthalmol, 104, 588, 10.1136/bjophthalmol-2019-314336 · DOI · Paper page
- Yang SH, Zhao HJ, Zhu GY, et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. 2024AAAI 2024;38(17):19368-19376. doi: 10.1609/aaai.v38i17.29907. · DOI · Paper page
Cited by
From language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50
“…workflows where they perform planning, decision making, and tool interaction in various real-world applications, including research assistance [6], software development [7], drug discoveries [8], multi-robot systems [9], clinical support [10], game simulation [11] and scientific simulations [12].”
Tzanis, E., Adams, L. C., Akinci D’Antonoli, T., Bressem, K. K., Cuocolo, R., Kocak, B., Malamateniou, C., & Klontzas, M. E. (2026) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2025.10.002 · Cited by 16
Jiang, X.-H., Yuan, X.-H., Zhao, H., & Peng, J.-S. (2025) · Journal of Medical Internet Research · doi:10.2196/76968 · Cited by 10
Agentic artificial intelligence is the future of cancer detection and diagnosis
Rahman, S., Hosain, M. T., Fahad, N., Morol, M. K., & Hossen, M. J. (2026) · Array · doi:10.1016/j.array.2025.100676 · Cited by 4
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
“However, prior healthcare multi-agent work largely centers on deliberation under the assumption that key evidence is already in-context, leaving evidence acquisition unsystematic (Chen et al., 2025; Gorenshtein et al., 2025; Wang et al., 2025).”
Metadata
- Authors
- Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D.
- Venue
- Intelligent Medicine
- Publisher
- Elsevier BV
- Published
- 2025-05
- Volume
- 5
- Issue
- 2
- Pages
- 151-163
- ISSN
- 2096-9376 (ISSN-L)
- 2667-1026
- OpenAlex
- W4409091111
- Cited by
- 68
- References
- 69
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Funders
- The Hong Kong Polytechnic University
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.