Evaluating the Performance of ChatGPT in Ophthalmology

May combine two claims.

ChatGPT Plus's weakest subjects were neuro-ophthalmology (40.0% ± 9.5%), oculoplastics (40.8% ± 9.2%), and clinical optics (45.8% ± 10.2%).

“However, its weakest subject remained neuro-ophthalmology (40.0% ± 9.5%) in addition to oculoplastics (40.8% ± 9.2%) and clinical optics (45.8% ± 10.2%).”

Probably not a substantial claim: may not pass S3 (load-bearing).

During experimentation, notable improvement in ChatGPT's performance was observed as the model was updated.

“During experimentation, we observed notable improvement in ChatGPT’s performance as the model was updated.”

May combine two claims.

With ChatGPT Plus, question difficulty (LR 32.30; P<0.001), examination section (LR 23.40; P=0.024), and cognitive level (LR 5.60; P=0.018) were all predictive of answer accuracy.

“With the improved ChatGPT Plus model, question difficulty (LR, 32.30; P < 0.001), followed by examination section (LR, 23.40; P = 0.024), and cognitive level (LR, 5.60; P = 0.018) were all predictive of ChatGPT Plus’s answer accuracy (Tables 3, S5 available at www.ophthalmologyscience.org).”

May combine two claims.

The legacy model performed best in general medicine and worst in neuro-ophthalmology (P<0.001) and ocular pathology (P=0.029).

“Although the legacy model performed best in general medicine and worst in neuro-ophthalmology (P < 0.001) and ocular pathology (P = 0.029), similar post hoc findings were not seen with ChatGPT Plus, suggesting more consistent results across examination sections.”

• Also stated in: § Discussion · plain

  “In contrast, the legacy model performed poorest in neuro-ophthalmology as well as ophthalmic pathology and intraocular tumors.”

May combine two claims. The quote only partly states this claim.

The BCSC and OphthoQuestions test sets had similar difficulty (P=0.154) and similar distribution of easy/moderate questions (P=0.102).

“The BCSC and OphthoQuestions training sets had a similar level of difficulty (P = 0.154) and mostly included easy and moderate questions in a very similar distribution (P = 0.102), as illustrated in Figure 1 and Table 1.”

May combine two claims.

ChatGPT Plus excelled in fundamentals (68.3% ± 6.1%), general medicine (65.8% ± 7.4%), and cornea (65.0% ± 3.2%).

“The updated ChatGPT Plus model consistently excelled in its strongest subjects: fundamentals (68.3% ± 6.1%), general medicine (65.8% ± 7.4%), and cornea (65.0% ± 3.2%).”

May combine two claims.

The legacy ChatGPT model achieved 55.8% accuracy on the BCSC set and 42.7% on the OphthoQuestions set.

“The legacy model achieved 55.8% accuracy on the BCSC set and 42.7% on the OphthoQuestions set.”

• Also stated in: § ChatGPT Had a Modest Overall Performance Initially but It Improved After a Model Update · plain

  “In the initial experiment, the legacy model achieved an accuracy of 55.8% on the BCSC set and 42.7% on the OphthoQuestions set.”

May combine two claims.

Taking data sets together, the legacy model performed well in general medicine (75%), fundamentals (60%), and cornea (60%), but not as well in neuro-ophthalmology (25%), glaucoma (37.5%), and pediatrics and strabismus (42.5%).

“Taking the data sets together, the legacy model performed well in general medicine (75%), fundamentals (60%), and cornea (60%), but not as well in neuro-ophthalmology (25%), glaucoma (37.5%), and pediatrics and strabismus (42.5%).”

May combine two claims.

Kappa values were 0.769 (95% CI 0.699-0.839) for BCSC and 0.798 (95% CI 0.728-0.868) for OphthoQuestions; combined kappa 0.786 (95% CI 0.736-0.835).

“The κ values were 0.769 (95% confidence interval [CI], 0.699–0.839) for the BCSC set and 0.798 (95% CI, 0.728–0.868) for the OphthoQuestions set. When these sets were combined, the resulting κ value was 0.786 (95% CI, 0.736–0.835).”

May combine two claims.

ChatGPT Plus provided highly consistent and repeatable results, but some variations occurred.

“Generally, we found that ChatGPT Plus provided highly consistent and repeatable results, but some variations occurred.”

May combine two claims.

On average, humans score 74% on the BCSC question bank; ophthalmology residents who completed training in 2022 obtained an average score of 63% on OphthoQuestions.

“The data indicate that, on average, humans score 74% on the BCSC question bank; and in the last 3 years, the group of ophthalmology residents who completed their training in 2022 obtained an average score of 63% on OphthoQuestions.”

May combine two claims.

The updated ChatGPT Plus model continued to perform poorly in neuro-ophthalmology, but its performance in pathology and intraocular tumors improved.

“The updated ChatGPT Plus model continued to perform poorly in neuro-ophthalmology, but its performance in pathology and intraocular tumors improved.”

May combine two claims.

ChatGPT Plus achieved 59.4% ± 0.6% accuracy on the BCSC set and 49.2% ± 1.0% on the OphthoQuestions set.

“With ChatGPT Plus, accuracy increased to 59.4% ± 0.6% and 49.2% ± 1.0%, respectively.”

• Also stated in: § ChatGPT Had a Modest Overall Performance Initially but It Improved After a Model Update · plain

  “However, with the improved ChatGPT Plus, the accuracy increased to 59.4% ± 0.6% on the BCSC set and 49.2% ± 1.0% on the OphthoQuestions set.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A new session was started in ChatGPT for each question to reduce memory retention bias.

“A new session was started in ChatGPT for each question to reduce memory retention bias.”

Probably not a substantial claim: may not pass S2 (checkable).

Large language models such as ChatGPT may need to incorporate other transformer models that can handle multiple types of data.

“Large language model, such as ChatGPT, may need to incorporate other transformer models that can handle multiple types of data, such as the Contrastive Language-Image Pretraining model,25 which can classify images and generate a text description that ChatGPT can then use to respond to a question.”

• cites [25] RadfordNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The degree of repeatability for runs was assessed within each examination section using a κ measure for M raters.

“The degree of repeatability for those runs was assessed within each examination section using a κ measure for M raters.20”

• cites [20] Fleiss, 1971, Measuring nominal scale agreement among many raters, Psychol Bull sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Similar referral patterns are observed in ocular oncology.

“For example, up to 40% of patients referred to a neuro-ophthalmology subspecialty service are misdiagnosed,23 and similar referral patterns are observed in ocular oncology.24”

• cites [24] Law, 2012, Referral patterns of intraocular tumour patients to a dedicated Canadian ocular oncology department, Can J Ophthalmol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Kung et al. showed that accuracy of ChatGPT is heavily influenced by concordance and insight, indicating inaccurate responses are caused by lack of training information for USMLE.

“In parallel, Kung et al11 showed that the accuracy of ChatGPT is heavily influenced by concordance and insight, indicating that inaccurate responses are caused by a lack of training information for the USMLE.”

• cites [11] Kung, 2023, Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models, PLOS Digit Health sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

We performed post hoc analysis using Tukey's test while controlling for question difficulty and cognitive level to isolate effect of examination section.

“We then performed a post hoc analysis using Tukey’s test to determine if there were significant differences in accuracy between examination sections while controlling for question difficulty and cognitive level. By controlling for those factors, we were able to isolate the effect of the examination section on accuracy and determine if there were any meaningful differences between the tested topics”

May combine two claims.

The questions in the test sets are not publicly accessible, excluding the possibility of prior indexing in search engines or ChatGPT training data.

“Those questions are not publicly accessible, thereby excluding the possibility of prior indexing in any search engine (like Google) or in the ChatGPT training data set.”

May combine two claims.

The BCSC and OphthoQuestions resources have been shown to be effective in studying for board examinations and linked to improved performance on OKAP.

“These resources have been shown to be effective in studying for board examinations and have been linked to improved performance on the standardized Ophthalmic Knowledge Assessment Program (OKAP) examination, which is taken annually by ophthalmology residents in the United States and Canada.12,13”

• cites [12] Zafar, 2019, Resident and program characteristics that impact performance on the Ophthalmic Knowledge Assessment Program (OKAP), BMC Med Educ sourceNot checked yet

  Not checked in the free run.

• cites [13] Lee, 2012, A multicenter analysis of the ophthalmic knowledge assessment program and American Board of Ophthalmology written qualifying examination performance, Ophthalmology sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Incorporating ChatGPT with other specialized foundation models trained on domain-specific sources might be required to improve its accuracy.

“Incorporating ChatGPT with other specialized foundation models that are trained using domain-specific sources (such as EyeWiki) might be required to improve its accuracy.”

May combine two claims.

The outcome is noteworthy and promising within ophthalmology, as results approach ChatGPT's performance on the USMLE despite being a much more specialized examination.

“Despite being slightly out of range of human performance, we believe that this outcome is noteworthy and promising within ophthalmology, as our results approach ChatGPT’s performance on the USMLE despite being a much more specialized examination.11”

• cites [11] Kung, 2023, Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models, PLOS Digit Health sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The model's high performance in general medicine, fundamentals, and cornea might be attributed to vast amount of training data and resources available on the internet for those topics.

“The model’s high performance in these areas might be attributed to the vast amount of training data and resources available on the internet for those topics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Up to 40% of patients referred to a neuro-ophthalmology subspecialty service are misdiagnosed.

“For example, up to 40% of patients referred to a neuro-ophthalmology subspecialty service are misdiagnosed,23 and similar referral patterns are observed in ocular oncology.24”

• cites [23] Stunkel, 2020, Referral patterns in neuro-ophthalmology, J Neuroophthalmol sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Kung et al. evaluated ChatGPT on USMLE multiple choice questions and found overall accuracy >50% in most experiments and provided insightful explanations.

“More recently, Kung et al11 evaluated the performance of ChatGPT, a generic LLM developed by OpenAI that is based on the GPT-3 series and optimized for dialogue, using multiple choice questions also from the USMLE. They found that ChatGPT achieved overall accuracy of > 50% in most of their experiments and also provided insightful explanations to support its answer choices.”

• cites [11] Kung, 2023, Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models, PLOS Digit Health sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The legacy model's accuracy was determined from a single run, whereas ChatGPT Plus means and SDs were derived from 3 runs.

“The legacy model’s accuracy was determined from a single run, whereas the means and standard deviations of the ChatGPT Plus model were derived from data collected over 3 runs.”

Classified as probably not a substantial claim. May combine two claims.

For ChatGPT Plus, we combined results of 3 runs and used a generalized estimating equation model with exchangeable correlation matrix and binomial distribution using logit link to account for correlated values.

“For the ChatGPT Plus model analyses, we combined the results of the 3 runs. Therefore, to account for correlated values, we used a generalized estimating equation model with an exchangeable correlation matrix and a binomial distribution using a logit link.”

Classified as probably not a substantial claim. May combine two claims.

We elected not to balance cognitive level and question difficulty manually to prevent cherry-picking, thereby avoiding bias.

“Despite having no control over the distribution of cognitive level and question difficulty in each of the randomly generated test sets, we elected not to balance them manually to prevent cherry-picking, thereby avoiding bias in the experiment results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ChatGPT is a fine-tuned LLM based on a model from GPT-3.5 series called 'gpt-3.5-turbo'.

“ChatGPT (OpenAI) is a fine-tuned LLM based on a model from the GPT-3.5 series called “gpt-3.5-turbo.”14”

• cites [14]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

We conducted ChatGPT Plus experiments thrice to ensure consistency of findings.

“Although we could not obtain repeat experiments on the legacy model, we conducted the ChatGPT Plus experiments thrice to ensure the consistency of our findings.”

Classified as probably not a substantial claim.

We used a simplified scoring system of low and high cognitive level to avoid making assumptions about intended goal of questions.

“We used a simplified scoring system of low and high cognitive level, instead of the 3-tier system proposed in the OKAP User’s Guide.18 This was done because we found it difficult to distinguish between level 2 and level 3 questions, and we wanted to avoid making assumptions about the intended goal of the questions.”

• cites [18] Americal Academy of Ophthalmology, OKAP Exam. Available at: https://www.aao.org/okap-exam. Published 2022. Accessed January 21, 2023.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Specializing LLMs through domain-specific pretraining may be necessary to improve their performance in ophthalmic subspecialties.

“Specializing LLMs through domain-specific pretraining may be necessary to improve their performance in ophthalmic subspecialties.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Singhal et al. demonstrated that Pathways Language Model achieved 67.6% accuracy on USMLE multiple choice questions.

“In 2022, Singhal et al9 demonstrated the capabilities of Pathways Language Model, a 540-billion parameter LLM, by testing it on multiple choice questions from the United States Medical Licensing Examination (USMLE) with an impressive 67.6% accuracy.”

• cites [9] Singhal, 2022, Large language models encode clinical knowledge, arXiv preprint arXiv:221213138Not checked yet

  Not checked in the free run.