Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Purpose: To evaluate the capabilities and incapabilities of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. Methods: We developed a digital ophthalmologist app using GPT-4V and evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit-lamp, scanning laser ophthalmoscopy, fundus photography of the posterior pole (FPP), optical coherence tomography, fundus fluorescein angiography and ocular ultrasound images. The chatbot was tested with ten open-ended questions per image, covering examination identification, lesion detection, diagnosis and decision support. The responses were manually assessed for accuracy, usability, safety and diagnosis repeatability. Auto-evaluation was performed using sentence similarity and GPT-4-based auto-evaluation. Results: Out of 600 responses, 30.6% were accurate, 21.5% were highly usable and 55.6% were deemed as no harm. GPT-4V performed best with slit-lamp images, with 42.0%, 38.5% and 68.5% of the responses being accurate, highly usable and no harm, respectively. However, its performance was weaker in FPP images, with only 13.7%, 3.7% and 38.5% in the same categories. GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracies in lesion identification (25.6%), diagnosis (16.1%) and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63.3% (38/60). The overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability. Conclusion: GPT-4V currently is not yet suitable for clinical decision-making in ophthalmology. Our study serves as a benchmark for enhancing ophthalmic multimodal models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the current capabilities of LLMs, including advanced models like GPT-4V, remain limited in their understanding and processing of medical images56,57”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [57]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Table 1 summarises the performance of GPT-4V in terms of accuracy, usability and safety based on the proportion of responses rated as good (score of 1) across 600 questions related to ophthalmic multimodal images.”
BackgroundSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“SLO, scanning laser ophthalmoscopy; FPP, fundus photography of the posterior pole; OCT, optical coherence tomography; FFA, fundus fluorescein angiography; OUS, ocular ultrasound.”
MethodLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“The image modalities and diagnoses are presented in online supplemental table 1.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 1×
“The image modalities and diagnoses are presented in online supplemental table 1.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“26 27 However, before applying any model to clinical practice, detailed clinical trials should be conducted to assess its safety and efficacy.”
MethodDave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · UCL Discovery (University College London) · Cited by 1,187
A foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014 · cited 1×
“24 Another way to improve the model is to align the ophthalmic image features with medical texts.”
BackgroundG-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · cited 2×
“An average of 25”
MethodG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · cited 2×
“An average of 25”
MethodMMBench: Is Your Multi-modal Model an All-Around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72658-3_13 · Cited by 668
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · doi:10.1101/2023.01.22.23284882 · Cited by 563
MMBench: Is Your Multi-modal Model an All-around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.06281 · Cited by 487 · cited 1×
“An average of 25”
GPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446 · cited 2×
“An average of 25”
MethodLim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C.-H., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 310
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K., Mansour, H. A., Abishek, R. M., Xu, D., Sridhar, J., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 178 · cited 1×
“These include answering questions in ophthalmology specialty exams such as OKAP and FRCOphth, 3 4 addressing surgical treatment-related queries in retinal diseases 5 and providing insights on myopia-related issues.”
BackgroundMedical visual question answering: A survey
Lin, Z., Zhang, D., Tao, Q., Shi, D., Haffari, G., Wu, Q., He, M., & Ge, Z. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102611 · Cited by 177
Self-citationGPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.365 · Cited by 166 · cited 2×
“An average of 25”
MethodNath, S., Marie, A., Ellershaw, S., Korot, E., & Keane, P. A. (2022) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2022-321141 · Cited by 116
Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images
Mihalache, A., Huang, R. S., Popovic, M. M., Patil, N. S., Pandya, B. U., Shor, R., Pereira, A., Kwok, J. M., Yan, P., Wong, D. T., et al. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.0017 · Cited by 105
Raimondi, R., Tzoumas, N., Salisbury, T., Di Simplicio, S., Romano, M. R., North East Trainee Research in Ophthalmology Network (NETRiON), Bommireddy, T., Chawla, H., Chen, Y., Connolly, S., et al. (2023) · Eye · doi:10.1038/s41433-023-02563-3 · Cited by 85
FFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72 · cited 1×
“An average of 25”
Self-citationA Comprehensive Study of GPT-4V’s Multimodal Capabilities in Medical ImagingPreprint
Li, Y., Liu, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2023) · medRxiv · doi:10.1101/2023.11.03.23298067 · Cited by 57 · cited 1×
“An average of 25”
Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model
Soong, D., Sridhar, S., Si, H., Wagner, J.-S., Sá, A. C. C., Yu, C. Y., Karagoz, K., Guan, M., Kumar, S., Hamadeh, H., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000568 · Cited by 49 · cited 1×
“An experienced ophthalmologist (PX) initially selected 60 images (representing 60 ophthalmic conditions across six modalities) from the aforementioned multimodal image dataset.”
MethodWang, C., Cheng, S., Guo, Q., Yue, Y., Ding, B., Xu, Z., Wang, Y., Hu, X., Zhang, Z., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12421 · Cited by 44 · cited 1×
“An average of 25”
Wu, C., Lei, J., Zheng, Q., Zhao, W., Lin, W., Zhang, X., Zhou, X., Zhao, Z., Zhang, Y., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.09909 · Cited by 43 · cited 1×
“An average of 25”
Self-citationA Survey on Hallucination in Large Vision-Language Models
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., & Peng, W. (2024) · arXiv · doi:10.48550/arxiv.2402.00253 · Cited by 37 · cited 1×
“26 27 However, before applying any model to clinical practice, detailed clinical trials should be conducted to assess its safety and efficacy.”
MethodICGA-GPT: report generation and question answering for indocyanine green angiography images
Chen, X., Zhang, W., Zhao, Z., Xu, P., Zheng, Y., Shi, D., & He, M. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324446 · Cited by 36
Self-citationChen, X., Xu, P., Li, Y., Zhang, W., Song, F., He, M., & Shi, D. (2024) · iScience · doi:10.1016/j.isci.2024.110021 · Cited by 32 · cited 1×
“11 25 Additionally, researchers may improve their models by applying transfer learning, fine-tuning and reinforcement learning techniques or by incorporating knowledge enrichment methods like retrieval-augmented generation.”
BackgroundSelf-citationPatient Questions and Physician Responses in a Chinese Health Q&A Website: Content Analysis
Hong, Z., Deng, Z., Evans, R. D., & Wu, H. (2020) · Journal of Medical Internet Research · doi:10.2196/13071 · Cited by 25 · cited 1×
“Due to the lack of universally recognised standard questions in the field of multimodal image-based question answering in ophthalmology, ten questions were constructed to evaluate the performance of GPT-4V based on previous studies 13 14 and the clinical experience of ophthalmologists.”
MethodWhat are Ophthalmology Patients Asking Online? An Analysis of the Eye Triage Subreddit
Mahjoub, H., Prabhu, A. V., & Sikder, S. (2020) · Clinical ophthalmology · doi:10.2147/opth.s279607 · Cited by 24
ChatFFA: Interactive Visual Question Answering on Fundus Fluorescein Angiography Image Using ChatGPTPreprint
Chen, X., Xu, P., Li, Y., Zhang, W., Song, F., Zheng, Y., Shi, D., & He, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4578568 · Cited by 6
Self-citation
Research works printed but not linked: 2
- Li Y , Wang L , Hu B , et al . A comprehensive evaluation of GPT-4V on knowledge-intensive visual question answering. arXiv 2311.07536 2023 [Preprint]. · arXiv
- Brooke, 1996, Sus: a “quick and Dirty’Usability, Usability Evaluation in Industry, 189, 189
Cited by
A multimodal visual–language foundation model for computational ophthalmology
Shi, D., Zhang, W., Yang, J., Huang, S., Chen, X., Xu, P., Jin, K., Lin, S. C., Wei, J., Yusufu, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01772-2 · Cited by 163
Self-citationEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 68
Self-citationFFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72
Self-citationLarge Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 79
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 73
Metadata
- Authors
- Xu, P., Chen, X., Zhao, Z., & Shi, D.
- Affiliations
- Hong Kong Polytechnic University (HK)
- Corresponding author
- Shi, Danli
- Venue
- British Journal of Ophthalmology
- Publisher
- BMJ
- Published
- 2024-05-24
- Volume
- 108
- Issue
- 10
- Pages
- 1384-1389
- ISSN
- 0007-1161 (ISSN-L)
- 1468-2079
- OpenAlex
- W4398775037
- Cited by
- 63
- References
- 26
- Citations by year
- 2026: 16 · 2025: 31 · 2024: 16
- FWCI
- 5.90
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- British Journal of Ophthalmology (publishedVersion)
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Funders
- Start-up Fund for RAPs under the Strategic Hiring Scheme (P0048623)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Keywords
- GPT-4V, large language models, ocular imaging, slit-lamp images, fundus photography, optical coherence tomography, fundus fluorescein angiography, ocular ultrasonography, lesion detection, clinical decision support, diagnostic accuracy, repeatability, safety assessment, sentence similarity
- MeSH terms
- Ophthalmologists, Diagnostic Techniques, Ophthalmological, Eye Diseases, Fluorescein Angiography, Humans, Ophthalmoscopy, Reproducibility of Results, Benchmarking, Tomography, Optical Coherence, Mobile Applications, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.