Comparison of Gemini Advanced and ChatGPT 4.0’s Performances on the Ophthalmology Resident Ophthalmic Knowledge Assessment Program (OKAP) Examination Review Question Banks
Open accessAbstract
Background With advancements in natural language processing, tools such as Chat Generative Pre-Trained Transformers (ChatGPT) version 4.0 and Google Bard's Gemini Advanced are being increasingly evaluated for their potential in various medical applications. The objective of this study was to systematically assess the performance of these language learning models (LLMs) on both image and non-image-based questions within the specialized field of Ophthalmology. We used a review question bank for the Ophthalmic Knowledge Assessment Program (OKAP) used by ophthalmology residents nationally to prepare for the Ophthalmology Board Exam to assess the accuracy and performance of ChatGPT and Gemini Advanced. Methodology A total of 260 randomly generated multiple-choice questions from the OphthoQuestions question bank were run through ChatGPT and Gemini Advanced. A simulated 260-question OKAP examination was created at random from the bank. Question-specific data were analyzed, including overall percent correct, subspecialty accuracy, whether the question was "high yield," difficulty (1-4), and question type (e.g., image, text). To compare the performance of ChatGPT-4 and Gemini on the difficulty of questions, we utilized the standard deviation of user answer choices to determine question difficulty. In this study, a statistical analysis of Google Sheets was conducted using two-tailed t-tests with unequal variance to compare the performance of ChatGPT-4.0 and Google's Gemini Advanced across various question types, subspecialties, and difficulty levels. Results In total, 259 of the 260 questions were included in the study as one question used a video that any form of ChatGPT could not interpret as of May 1, 2024. For text-only questions, ChatGPT-4.0.0 correctly answered 57.14% (148/259, p < 0.018), and Gemini Advanced correctly answered 46.72% (121/259, p < 0.018). Both versions answered most questions without a prompt and would have received a below-average score on the OKAP. Moreover, there were 27 questions utilizing a secondary prompt in ChatGPT-4.0 compared to 67 questions in Gemini Advanced. ChatGPT-4.0 performed 68.99% on easier questions (<2 on a scale from 1-4) and 44.96% on harder questions (>2 on a scale from 1-4). On the other hand, Gemini Advanced performed 49.61% on easier questions (<2 on a scale from 1-4) and 44.19% on harder questions (>2 on a scale from 1-4). There was a statistically significant difference in accuracy between ChatGPT-4.0 and Gemini Advanced for easy (p < 0.0015) but not for hard (p < 0.55) questions. For image-only questions, ChatGPT-4.0 correctly answered 39.58% (19/48, p < 0.013), and Gemini Advanced correctly answered 33.33% (16/48, p < 0.022), resulting in a statistically insignificant difference in accuracy between ChatGPT-4.0 and Gemini Advanced (p < 0.530). A comparison against text-only and image-based questions demonstrated a statistically significant difference in accuracy for both ChatGPT-4.0 (p < 0.013) and Gemini Advanced (p < 0.022). Conclusions This study provides evidence that ChatGPT-4.0 performs better on the OKAP-style exams and is improved compared to Gemini Advanced within the context of ophthalmic multiple-choice questions. This may show an opportunity for increased worth for ChatGPT in ophthalmic medical education. While showing promise within medical education, caution should be used as a more detailed evaluation of reliability is needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 2× · see citing works
“In doing so, LLMs help to ensure that questions test the intended knowledge and skills rather than a student’s ability to interpret confusing or poorly structured language [19].”
BackgroundGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J. P., & Lim, C. C. T. (2017) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,521 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414 · see citing works
Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J., Shin, J. H., Sullivan, P. L. Z., Cielo, D., Oyelese, A. A., Doberstein, C. E., Telfeian, A. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002551 · Cited by 354 · see citing works
Homolak, J. (2023) · Croatian Medical Journal · doi:10.3325/cmj.2023.64.1 · Cited by 208 · see citing works
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195 · see citing works
Large language models for generating medical examinations: systematic review
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05239-y · Cited by 128 · see citing works
Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., & Takase, K. (2023) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01491-2 · Cited by 126 · see citing works
Guerra, G. A., Hofmann, H., Sobhani, S., Hofmann, G., Gomez, D., Soroudi, D., Hopkins, B. S., Dallas, J., Pangal, D. J., Cheok, S., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.08.042 · Cited by 123 · see citing works
Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment
Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231193716 · Cited by 101 · see citing works
ChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100 · see citing works
Carlà, M. M., Gambini, G., Baldascino, A., Giannuzzi, F., Boselli, F., Crincoli, E., D’Onofrio, N. C., & Rizzo, S. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325143 · Cited by 97 · see citing works
Improved Performance of ChatGPT-4 on the OKAP Examination: A Comparative Study with ChatGPT-3.5
Teebagy, S., Colwell, L., Wood, E., Yaghy, A., & Faustina, M. (2023) · Journal of Academic Ophthalmology · doi:10.1055/s-0043-1774399 · Cited by 86 · see citing works
Antaki, F., Milad, D., Chia, M. A., Giguère, C., Touma, S., El‐Khoury, J., Keane, P. A., & Duval, R. (2023) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324438 · Cited by 83 · see citing works
Carlà, M. M., Gambini, G., Baldascino, A., et al. (2024) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-024-06470-5 · Cited by 83 · see citing works
Mihalache, A., Grad, J., Patil, N. S., Huang, R. S., Popovic, M. M., Mallipatna, A. C., Kertes, P. J., & Muni, R. H. (2024) · Eye · doi:10.1038/s41433-024-03067-4 · Cited by 50 · see citing works
Frosolini, A., Catarzi, L., Benedetti, S., Latini, L., Chisci, G., Franz, L., Gennaro, P., & Gabriele, G. (2024) · Diagnostics · doi:10.3390/diagnostics14080839 · Cited by 41 · see citing works
Şahin, M. Ç., Sözer, A., Kuzucu, P., Türkmen, T., Şahin, M. B., Sozer, E., Tüfek, O. Y., Nernekli, K., Emmez, H., & Çeltikçi, E. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107807 · Cited by 37 · see citing works
Botross, M., Mohammadi, S. O., Montgomery, K. E., & Crawford, C. M. (2024) · Cureus · doi:10.7759/cureus.57348 · Cited by 32 · see citing works
Shukla, R., Mishra, A. K., Banerjee, N., & Verma, A. (2024) · Cureus · doi:10.7759/cureus.58232 · Cited by 31 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Karataş, G., & Karataş, M. E. (2026) · Strabismus · doi:10.1080/09273972.2025.2536782 · Cited by 7 · see citing works
Yang, M., Long, J., He, Y., Meng, R., Ling, C., Long, Q., & Wen, J. (2026) · BMC Nursing · doi:10.1186/s12912-026-05132-7
“0, Gemini Advanced, and DeepSeek, for instance, have shown promising results on ophthalmology knowledge tests and clinical reasoning exercises, with certain models outperforming others in terms of reasoning accuracy, factual correctness, and consistency [22, 31].”
- Authors
- Gill, G. S., Tsai, J., Moxam, J., Sanghvi, H. A., & Gupta, S.
- Affiliations
- Florida Atlantic University (US)
- Broward Health Medical Center (US)
- University of Florida (US)
- Corresponding author
- Gill, Gurnoor S
- Venue
- Cureus
- Published
- 2024-09-17
- Volume
- 16
- Issue
- 9
- Pages
- e69612-e69612
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC11486483
- OpenAlex
- W4402567360
- Cited by
- 22
- References
- 23
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 1
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT-4, Gemini Advanced, large language models, image-based questions, ophthalmology residents, AI performance evaluation, medical education technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
