Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions
Open accessAbstract
Importance Chat Generative Pre-Trained Transformer (ChatGPT) has shown promising performance in various fields, including medicine, business, and law, but its accuracy in specialty-specific medical questions, particularly in ophthalmology, is still uncertain. Purpose This study evaluates the performance of two ChatGPT models (GPT-3.5 and GPT-4) and human professionals in answering ophthalmology questions from the StatPearls question bank, assessing their outcomes, and providing insights into the integration of artificial intelligence (AI) technology in ophthalmology. Methods ChatGPT's performance was evaluated using 467 ophthalmology questions from the StatPearls question bank. These questions were stratified into 11 subcategories, four difficulty levels, and three generalized anatomical categories. The answer accuracy of GPT-3.5, GPT-4, and human participants was assessed. Statistical analysis was conducted via the Kolmogorov-Smirnov test for normality, one-way analysis of variance (ANOVA) for the statistical significance of GPT-3 versus GPT-4 versus human performance, and repeated unpaired two-sample t-tests to compare the means of two groups. Results GPT-4 outperformed both GPT-3.5 and human professionals on ophthalmology StatPearls questions, except in the "Lens and Cataract" category. The performance differences were statistically significant overall, with GPT-4 achieving higher accuracy (73.2%) compared to GPT-3.5 (55.5%, p-value < 0.001) and humans (58.3%, p-value < 0.001). There were variations in performance across difficulty levels (rated one to four), but GPT-4 consistently performed better than both GPT-3.5 and humans on level-two, -three, and -four questions. On questions of level-four difficulty, human performance significantly exceeded that of GPT-3.5 (p = 0.008). Conclusion The study's findings demonstrate GPT-4's significant performance improvements over GPT-3.5 and human professionals on StatPearls ophthalmology questions. Our results highlight the potential of advanced conversational AI systems to be utilized as important tools in the education and practice of medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The obstacle of AI chatbot hallucination is improving gradually, especially through research, dedicated medical chatbots, and upgrading existing ones.
“However, it is worth noting that this obstacle is improving gradually, especially by introducing research, dedicated medical chatbots, and upgrading existing ones [16,30].”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [30]
The Discussion reports substantially better ophthalmology question-answering performance for GPT-4 than GPT-3.5, supporting the point that upgrading chatbots can improve accuracy. It does not establish that hallucinations are gradually declining or that dedicated medical chatbots are responsible for such improvement. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- [2]Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models PLOS Digit Health Kung TH Cheatham M Medenilla A 0 2 2023 PMC9931230 36812645
- [3]ChatGPT goes to law school J Leg Educ Choi JH Hickman KE Monahan A [Epub] 2023 https://deliverypdf.ssrn.com/delivery.php?ID=631024088114108115116064085097081030097033001017049013093124093124025103066031089067106004003004103059109070004073116115065075019040011087093004025030095079088006026049041083093100083019072022119121107121010027108106004111074080087096114028089012125071&EXT=pdf&INDEX=TRUE
- [5]Performance of ChatGPT on the plastic surgery inservice training examination Aesthet Surg J Gupta R Herzog I Park JB 0 2023 37128784
- [6]ChatGPT fails the multiple-choice American College of Gastroenterology self-assessment test Am J Gastroenterol Suchman K Garg S Trindade A [Online ahead of press] 2023 https://pubmed.ncbi.nlm.nih.gov/37212584/ 37212584
- [10]Evaluating ChatGPT as an adjunct for radiologic decision-making (Preprint) MedRxiv Rao A Kim J Kamineni M Pang M Lie W Succi MD 2023 https://doi.org/10.1101/2023.02.02.23285399 PMC10733745 37356806 · DOI · Paper page
- [11]Diagnostic accuracy of differential-diagnosis lists generated by Generative Pretrained Transformer 3 chatbot for clinical vignettes with common chief complaints: a pilot study Int J Environ Res Public Health Hirosawa T Harada Y Yokose M Sakamoto T Kawamura R Shimizu T 3378 20 2023 https://doi.org/10.3390/ijerph20043378 36834073 PMC9967747 · DOI
- [12]Large language model (ChatGPT) as a support tool for breast tumor board NPJ Breast Cancer Sorin V Klang E Sklair-Levy M 44 9 2023 37253791 PMC10229606
- [13]Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment JAMA Ophthalmol Mihalache A Popovic MM Muni RH 589 597 141 2023 37103928 PMC10141269
- [14]ChatGPT takes on the European Exam in core cardiology: an artificial intelligence success story? Eur Heart J Digit Health Skalidis I Cagnina A Luangphiphat W Mahendiran T Muller O Abbe E Fournier S 279 281 4 2023 37265864 PMC10232281
- [15]Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT Clin Orthop Relat Res Lum ZC 2023 PMC10344569 37220190
- [16]Improved performance of ChatGPT-4 on the OKAP exam: a comparative study with ChatGPT-3.5 (Preprint) MedRxiv Teebagy S Colwell L Wood E Yaghy A Faustina M 2023 PMC10495224 37701862
- [17]Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings Ophthalmol Sci Antaki F Touma S Milad D El-Khoury J Duval R 100324 3 2023 37334036 PMC10272508
- [18]Performance of ChatGPT on dermatology Specialty Certificate Examination multiple choice questions Clin Exp Dermatol Passby L Jenko N Wernham A 0 2023 37264670
Cited by
Sorted by citations
Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges
Zhang, P., & Kamel Boulos, M. N. (2023) · Future Internet · doi:10.3390/fi15090286 · Cited by 411 · see citing works
“The more recent a model’s version is, the better it tends to perform, e.g., [60], but this is not always guaranteed [61].”
BackgroundLarge Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201 · see citing works
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024) · JMIR Medical Informatics · doi:10.2196/54345 · Cited by 166 · see citing works
“However, it is worth noting that this obstacle is improving gradually, especially by introducing research, dedicated medical chatbots, and upgrading existing ones [16,30].”
BackgroundChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146 · see citing works
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69 · see citing works
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49 · see citing works
Mondal, H., Karri, J. K. K., Ramasubramanian, S., Mondal, S., Juhi, A., & Gupta, P. (2025) · Advances in Physiology Education · doi:10.1152/advan.00088.2024 · Cited by 28 · see citing works
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
Karataş, G., & Karataş, M. E. (2026) · Strabismus · doi:10.1080/09273972.2025.2536782 · Cited by 7 · see citing works
- Authors
- Moshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C.
- Affiliations
- The University of Texas at San Antonio Health Science Center (US)
- Corresponding author
- Moshirfar, Majid
- Venue
- Cureus
- Published
- 2023-06-22
- Volume
- 15
- Issue
- 6
- Pages
- e40822-e40822
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC10362981
- OpenAlex
- W4381743186
- Cited by
- 124
- References
- 1
- Citations by year
- 2026: 4 · 2025: 43 · 2024: 62 · 2023: 15
- FWCI
- 4.95
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, GPT-4, GPT-3.5, large language models, ophthalmology, question bank, answer accuracy, question difficulty, medical education, conversational AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
