CoChat

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Open access
Antaki, Fares, Touma, Samir, Milad, Daniel, El-Khoury, Jonathan, Duval, Renaud · 2023 · Ophthalmology Science

Preprint — published as 10.1016/j.xops.2023.100324 · Paper page

The DOI matches its Crossref record; its reference list is not available.

Abstract

We tested the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space using two popular multiple choice question banks used for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) exam. The testing sets were of easy-to-moderate difficulty and were diversified, including recall, interpretation, practical and clinical decision-making problems. ChatGPT achieved 55.8% and 42.7% accuracy in the two 260-question simulated exams. Its performance varied across subspecialties, with the best results in general medicine and the worst in neuro-ophthalmology and ophthalmic pathology and intraocular tumors. These results are encouraging but suggest that specialising LLMs through domain-specific pre-training may be necessary to improve their performance in ophthalmic subspecialties.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.01.22.23284882

Cites

Statistics

Figures

Figure 1 Alluvial diagram illustrating the distribution of questions across examination sections, cognitive level, and question difficulty. Despite having been generated at random, the Basic and Clinical Science Course (BCSC) and OphthoQuestions test sets have a similar distribution of questions with high and low cognitive levels and similar difficulty.
Figure 1 Alluvial diagram illustrating the distribution of questions across examination sections, cognitive level, and question difficulty. Despite having been generated at random, the Basic and Clinical Science Course (BCSC) and OphthoQuestions test sets have a similar distribution of questions with high and low cognitive levels and similar difficulty.CC BY-NC-ND · PubMed Central
Figure 2 Bar plot of the accuracy of ChatGPT across examination sections and ChatGPT models for the Basic and Clinical Science Course (BCSC) and OphthoQuestions testing sets. The ChatGPT Plus model accuracy is shown with error bars representing the standard deviation from the 3 experimental runs.
Figure 2 Bar plot of the accuracy of ChatGPT across examination sections and ChatGPT models for the Basic and Clinical Science Course (BCSC) and OphthoQuestions testing sets. The ChatGPT Plus model accuracy is shown with error bars representing the standard deviation from the 3 experimental runs.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 23 · Cited by 563

Research works printed but not linked: 5

  1. 10.1136/bjophthalmol-2018-313173 · DOI · Paper page
  2. Bommasani R , Hudson DA , Adeli E , et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:210807258 2021
  3. Singhal K , Azizi S , Tu T , et al. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:221213138 2022
  4. Liévin V , Hother CE , Winther O. Can large language models reason about medical questions? arXiv preprint arXiv:220708143 2022
  5. Ouyang L , Wu J , Jiang X , et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:220302155 2022

Metadata

Authors
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R.
Affiliations
Hôpital Maisonneuve-Rosemont (CA)
Corresponding author
Duval, Renaud
Venue
Ophthalmology Science
Publisher
openRxiv
Published
2023-01-24
Volume
3
Issue
4
Pages
100324-100324
ISSN
2666-9145 (ISSN-L)
OpenAlex
W4368340908
Cited by
563
References
23
Citations by year
2026: 69 · 2025: 145 · 2024: 201 · 2023: 148
FWCI
22.16
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Ophthalmology Science (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion, cc-by-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Bayer
  • American Academy of Ophthalmology
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
ChatGPT, large language models, foundation models, answer accuracy, question difficulty, cognitive levels, neuro-ophthalmology, ocular pathology, ChatGPT Plus, domain-specific pre-training, ophthalmology education, artificial intelligence in medical education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings. https://app.cochat.ai/papers/10.1101/2023.01.22.23284882
The paper
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023). Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings. Ophthalmology Science. https://doi.org/10.1101/2023.01.22.23284882

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.