Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis

Open access
Xu, Pusheng, Chen, Xiaolan, Zhao, Ziwei, Shi, Danli·2024·British Journal of Ophthalmology

Published version of a preprint

Abstract

Purpose: To evaluate the capabilities and incapabilities of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. Methods: We developed a digital ophthalmologist app using GPT-4V and evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit-lamp, scanning laser ophthalmoscopy, fundus photography of the posterior pole (FPP), optical coherence tomography, fundus fluorescein angiography and ocular ultrasound images. The chatbot was tested with ten open-ended questions per image, covering examination identification, lesion detection, diagnosis and decision support. The responses were manually assessed for accuracy, usability, safety and diagnosis repeatability. Auto-evaluation was performed using sentence similarity and GPT-4-based auto-evaluation. Results: Out of 600 responses, 30.6% were accurate, 21.5% were highly usable and 55.6% were deemed as no harm. GPT-4V performed best with slit-lamp images, with 42.0%, 38.5% and 68.5% of the responses being accurate, highly usable and no harm, respectively. However, its performance was weaker in FPP images, with only 13.7%, 3.7% and 38.5% in the same categories. GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracies in lesion identification (25.6%), diagnosis (16.1%) and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63.3% (38/60). The overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability. Conclusion: GPT-4V currently is not yet suitable for clinical decision-making in ophthalmology. Our study serves as a benchmark for enhancing ophthalmic multimodal models.

Overview

Integrity signals

Registry

Crossref · 10.1136/bjo-2023-325054

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 26 references

Research works printed but not linked: 4

  1. Li Y , Wang L , Hu B , et al . A comprehensive evaluation of GPT-4V on knowledge-intensive visual question answering. arXiv 2311.07536 2023 [Preprint]. · arXiv
  2. Brooke, 1996, Sus: a “quick and Dirty’Usability, Usability Evaluation in Industry, 189, 189
  3. Wang C , Cheng S , Xu Z , et al . Evaluating open question answering evaluation. arXiv 2305.12421 [Preprint] 2023. · arXiv
  4. Liu H , Xue W , Chen Y , et al . A survey on hallucination in large vision-language models. arXiv Preprint arXiv:2402.00253 2024. arXiv 2402.00253 [Preprint] 2024. · arXiv

Cited by

See all 63

Metadata

Authors
Xu, P., Chen, X., Zhao, Z., & Shi, D.
Affiliations
Hong Kong Polytechnic University (HK)
Corresponding author
Shi, Danli
Venue
British Journal of Ophthalmology
Publisher
BMJ
Published
2024-05-24
Volume
108
Issue
10
Pages
1384-1389
ISSN
  • 0007-1161 (ISSN-L)
  • 1468-2079
OpenAlex
W4398775037
Cited by
63
References
26
Citations by year
2026: 16 · 2025: 31 · 2024: 16
FWCI
5.90
Type
journal article
Language
en
Open access
Locations
  • British Journal of Ophthalmology (publishedVersion)
  • medRxiv (acceptedVersion)
  • PubMed (publishedVersion)
  • PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
Funders
Start-up Fund for RAPs under the Strategic Hiring Scheme (P0048623)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
Keywords
GPT-4V, large language models, ocular imaging, slit-lamp images, fundus photography, optical coherence tomography, fundus fluorescein angiography, ocular ultrasonography, lesion detection, clinical decision support, diagnostic accuracy, repeatability, safety assessment, sentence similarity
MeSH terms
Ophthalmologists, Diagnostic Techniques, Ophthalmological, Eye Diseases, Fluorescein Angiography, Humans, Ophthalmoscopy, Reproducibility of Results, Benchmarking, Tomography, Optical Coherence, Mobile Applications, Multimodal Imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. https://app.cochat.ai/papers/10.1136/bjo-2023-325054
The paper
Xu, P., Chen, X., Zhao, Z., & Shi, D. (2024). Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. British Journal of Ophthalmology. https://doi.org/10.1136/bjo-2023-325054

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.