The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis
Open accessPublished version of a preprint
Abstract
Purpose: To evaluate the capabilities and incapabilities of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. Methods: We developed a digital ophthalmologist app using GPT-4V and evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit-lamp, scanning laser ophthalmoscopy, fundus photography of the posterior pole (FPP), optical coherence tomography, fundus fluorescein angiography and ocular ultrasound images. The chatbot was tested with ten open-ended questions per image, covering examination identification, lesion detection, diagnosis and decision support. The responses were manually assessed for accuracy, usability, safety and diagnosis repeatability. Auto-evaluation was performed using sentence similarity and GPT-4-based auto-evaluation. Results: Out of 600 responses, 30.6% were accurate, 21.5% were highly usable and 55.6% were deemed as no harm. GPT-4V performed best with slit-lamp images, with 42.0%, 38.5% and 68.5% of the responses being accurate, highly usable and no harm, respectively. However, its performance was weaker in FPP images, with only 13.7%, 3.7% and 38.5% in the same categories. GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracies in lesion identification (25.6%), diagnosis (16.1%) and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63.3% (38/60). The overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability. Conclusion: GPT-4V currently is not yet suitable for clinical decision-making in ophthalmology. Our study serves as a benchmark for enhancing ophthalmic multimodal models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the current capabilities of LLMs, including advanced models like GPT-4V, remain limited in their understanding and processing of medical images56,57”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [57]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · UCL Discovery (University College London) · Cited by 1,187
A foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
Research works printed but not linked: 4
- Li Y , Wang L , Hu B , et al . A comprehensive evaluation of GPT-4V on knowledge-intensive visual question answering. arXiv 2311.07536 2023 [Preprint]. · arXiv
- Brooke, 1996, Sus: a “quick and Dirty’Usability, Usability Evaluation in Industry, 189, 189
- Wang C , Cheng S , Xu Z , et al . Evaluating open question answering evaluation. arXiv 2305.12421 [Preprint] 2023. · arXiv
- Liu H , Xue W , Chen Y , et al . A survey on hallucination in large vision-language models. arXiv Preprint arXiv:2402.00253 2024. arXiv 2402.00253 [Preprint] 2024. · arXiv
Cited by
A multimodal visual–language foundation model for computational ophthalmology
Shi, D., Zhang, W., Yang, J., Huang, S., Chen, X., Xu, P., Jin, K., Lin, S. C., Wei, J., Yusufu, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01772-2 · Cited by 132
Self-citationEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 100
Self-citationFFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72
Self-citationLarge Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 70
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 64
Metadata
- Authors
- Xu, P., Chen, X., Zhao, Z., & Shi, D.
- Affiliations
- Hong Kong Polytechnic University (HK)
- Corresponding author
- Shi, Danli
- Venue
- British Journal of Ophthalmology
- Publisher
- BMJ
- Published
- 2024-05-24
- Volume
- 108
- Issue
- 10
- Pages
- 1384-1389
- ISSN
- 0007-1161 (ISSN-L)
- 1468-2079
- OpenAlex
- W4398775037
- Cited by
- 63
- References
- 26
- Citations by year
- 2026: 16 · 2025: 31 · 2024: 16
- FWCI
- 5.90
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- British Journal of Ophthalmology (publishedVersion)
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Funders
- Start-up Fund for RAPs under the Strategic Hiring Scheme (P0048623)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Keywords
- GPT-4V, large language models, ocular imaging, slit-lamp images, fundus photography, optical coherence tomography, fundus fluorescein angiography, ocular ultrasonography, lesion detection, clinical decision support, diagnostic accuracy, repeatability, safety assessment, sentence similarity
- MeSH terms
- Ophthalmologists, Diagnostic Techniques, Ophthalmological, Eye Diseases, Fluorescein Angiography, Humans, Ophthalmoscopy, Reproducibility of Results, Benchmarking, Tomography, Optical Coherence, Mobile Applications, Multimodal Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. https://app.cochat.ai/papers/10.1136/bjo-2023-325054
- The paper
- Xu, P., Chen, X., Zhao, Z., & Shi, D. (2024). Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. British Journal of Ophthalmology. https://doi.org/10.1136/bjo-2023-325054
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.