Performance of an upgraded artificial intelligence chatbot for ophthalmic knowledge assessment.
Open accessAbstract
This cross-sectional study assesses the accuracy of answers generated by an updated version of a popular chatbot to board certification examination preparation questions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Across other medical subspecialties, proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly, while proprietary LLM 2 answered 57% to 84% correctly.
“This improvement may explain some of the variation in performance observed across examinations from other medical subspecialties, where proprietary LLM 1 and other LLMs answered 45% to 77% of questions correctly,6,7,8,9,11,35 while proprietary LLM 2 answered 57% to 84% correctly.10,12”
From Performance of Large Language Models on Medical Oncology Examination Questions (2024) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ChatGPT User Experience: Implications for Education
Zhai, X. (2023) · doi:10.2139/ssrn.4312418 · Cited by 935 · see citing works
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 305 · see citing works
Self-citationGPT-4 is here: what scientists think
Sanderson, K. (2023) · Nature · doi:10.1038/d41586-023-00816-5 · Cited by 262 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [4]OpenAI. GPT-4 technical report. Preprint posted online March 2023. https://arxiv.org/abs/2303.08774 · arXiv
Cited by
Sorted by citations
ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis
Bicknell, B. T., Butler, D., Whalen, S., Ricks, J., Dixon, C. J., Clark, A. B., Spaedy, O., Skelton, A., Edupuganti, N., Dzubinski, L., et al. (2024) · JMIR Medical Education · doi:10.2196/63430 · Cited by 107 · see citing works
Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249588 · Cited by 112 · see citing works
Self-citationAccuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images
Mihalache, A., Huang, R. S., Popovic, M. M., Patil, N. S., Pandya, B. U., Shor, R., Pereira, A., Kwok, J. M., Yan, P., Wong, D. T., et al. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.0017 · Cited by 91 · see citing works
Self-citationPerformance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Danesh, A., Pazouki, H., Danesh, F., Danesh, A., & Vardar‐Şengül, S. (2024) · Journal of Periodontology · doi:10.1002/jper.23-0514 · Cited by 67 · see citing works
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 61 · see citing works
What can GPT-4 do for Diagnosing Rare Eye Diseases? A Pilot Study
Hu, X., Ran, A. R., Nguyen, T. X., Szeto, S., Yam, J. C. S., Chan, C. K. M., & Cheung, C. Y.-L. (2023) · Ophthalmology and Therapy · doi:10.1007/s40123-023-00789-8 · Cited by 52 · see citing works
Mihalache, A., Grad, J., Patil, N. S., Huang, R. S., Popovic, M. M., Mallipatna, A. C., Kertes, P. J., & Muni, R. H. (2024) · Eye · doi:10.1038/s41433-024-03067-4 · Cited by 50 · see citing works
Self-citationA descriptive study based on the comparison of ChatGPT and evidence-based neurosurgeons
Liu, J., Zheng, J., Cai, X., Wu, D., & Yin, C. (2023) · iScience · doi:10.1016/j.isci.2023.107590 · Cited by 44 · see citing works
Ittarat, M., Cheungpasitporn, W., & Chansangpetch, S. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13121679 · Cited by 43 · see citing works
Strzalkowski, P., Strzalkowska, A., Chhablani, J. K., Pfau, K., Errera, M., Roth, M., Schaub, F., Bechrakis, N. E., Hoerauf, H., Reiter, C., et al. (2024) · International Journal of Retina and Vitreous · doi:10.1186/s40942-024-00579-9 · Cited by 44 · see citing works
Sevgi, M., Antaki, F., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325046 · Cited by 40 · see citing works
A survey for large language models in biomedicine
Wang, C., Li, M., He, J., Wang, Z., Darzidehkalani, E., Chen, Z., Ye, J., Li, T., Su, Y., Ke, J., et al. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103268 · Cited by 51 · see citing works
Evaluating Large Language Models in Ophthalmology: Systematic Review
Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025) · Journal of Medical Internet Research · doi:10.2196/76947 · Cited by 35 · see citing works
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 35 · see citing works
Tao, B. K.-L., Hua, N. J., Milkovich, J., & Micieli, J. A. (2024) · Eye · doi:10.1038/s41433-024-03037-w · Cited by 32 · see citing works
ChatGPT giving advice on how to cheat in university assignments: how workable are its suggestions?
Spennemann, D. H. R., Biles, J. M., Brown, L., Ireland, M. F., Longmore, L., Singh, C., Wallis, A., & Ward, C. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-10-2023-0195 · Cited by 29 · see citing works
Performance of ChatGPT in Ophthalmic Registration and Clinical Diagnosis: Cross-Sectional Study
Ming, S., Yao, X., Guo, X., Guo, Q., Xie, K., Chen, D., & Lei, B. (2024) · Journal of Medical Internet Research · doi:10.2196/60226 · Cited by 23 · see citing works
Jiao, C., Rosas, E., Asadigandomani, H., Delsoz, M., Madadi, Y., Raja, H., Munir, W. M., Tamm, B., Mehravaran, S., Djalilian, A. R., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15101221 · Cited by 23 · see citing works
Wei, J., Wang, X., Huang, M., Xu, Y., & Yang, W. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02227-7 · Cited by 23 · see citing works
Ming, S., Guo, Q., Cheng, W., & Lei, B. (2024) · JMIR Medical Education · doi:10.2196/52784 · Cited by 25 · see citing works
Large Language Models in Ophthalmology: A Review of Publications from Top Ophthalmology Journals
Agnihotri, A. P., Nagel, I. D., Artiaga, J. C. M., Guevarra, M. C. B., Sosuan, G. M. N., & Kalaw, F. G. P. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100681 · Cited by 21 · see citing works
Gritti, M. N., AlTurki, H., Farid, P., & Morgan, C. T. (2024) · Pediatric Cardiology · doi:10.1007/s00246-023-03385-6 · Cited by 17 · see citing works
Marshall, R. F., Mallem, K., Xu, H., Thorne, J. E., Burkholder, B. M., Chaon, B., Liberman, P., & Berkenstock, M. (2024) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2024.2317417 · Cited by 19 · see citing works
Interpretation of Clinical Retinal Images Using an Artificial Intelligence Chatbot
Mihalache, A., Huang, R. S., Mikhail, D., Popovic, M. M., Shor, R., Pereira, A., Kwok, J. M., Yan, P., Wong, D. T., Kertes, P. J., et al. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100556 · Cited by 19 · see citing works
Self-citationUtilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges
Chotcomwongse, P., Ruamviboonsuk, P., & Grzybowski, A. E. (2024) · Ophthalmology and Therapy · doi:10.1007/s40123-024-01018-6 · Cited by 19 · see citing works
Performance of ChatGPT in Israeli Hebrew OBGYN national residency examinations
Cohen, A., Alter, R., Lessans, N., Meyer, R., Brezinov, Y., & Levin, G. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07185-4 · Cited by 17 · see citing works
Wang, D., Ye, J., Li, J., Liang, J., Zhang, Q., Hu, Q., Pan, C., Wang, D., Liu, Z. Y., Shi, W. H., et al. (2025) · JMIR Medical Education · doi:10.2196/70190 · Cited by 15 · see citing works
PERFORMANCE ASSESSMENT OF AN ARTIFICIAL INTELLIGENCE CHATBOT IN CLINICAL VITREORETINAL SCENARIOS
Maywood, M. J., Parikh, R., Deobhakta, A. A., & Begaj, T. (2024) · Retina · doi:10.1097/iae.0000000000004053 · Cited by 15 · see citing works
AI in Neuro-Ophthalmology: Current Practice and Future Opportunities
Kenney, R., Requarth, T., Jack, A. I., Hyman, S., Galetta, S. L., & Grossman, S. N. (2024) · Journal of Neuro-Ophthalmology · doi:10.1097/wno.0000000000002205 · Cited by 15 · see citing works
Chatbot and Academy Preferred Practice Pattern Guidelines on Retinal Diseases
Mihalache, A., Huang, R. S., Patil, N. S., Popovic, M. M., Lee, W. W., Yan, P., Cruz-Pimentel, M., & Muni, R. H. (2024) · Ophthalmology Retina · doi:10.1016/j.oret.2024.03.013 · Cited by 13 · see citing works
Self-citationPerformance of Language Models on the Family Medicine In-Training Exam
Hanna, R. E., Smith, L. R., Mhaskar, R. S., & Hanna, K. (2024) · Family Medicine · doi:10.22454/fammed.2024.233738 · Cited by 13 · see citing works
Artificial intelligence in ophthalmology
Wiedemann, P. (2023) · International Journal of Ophthalmology · doi:10.18240/ijo.2023.09.01 · Cited by 12 · see citing works
Ethical considerations for large language models in ophthalmology
Kalaw, F. G. P., & Baxter, S. L. (2024) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000001083 · Cited by 12 · see citing works
ChatGPT and frequently asked patient questions for upper eyelid blepharoplasty surgery
Watane, A., Perzia, B., Weiss, M. E., Tooley, A. A., Li, E., Habib, L. A., Tenzel, P. A., & Maeng, M. M. (2024) · Orbit · doi:10.1080/01676830.2024.2435930 · Cited by 9 · see citing works
Olis, M., Dyjak, P. M., & Weppelmann, T. A. (2024) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2024.01.011 · Cited by 8 · see citing works
Artificial intelligence chatbot interpretation of ophthalmic multimodal imaging cases
Mihalache, A., Huang, R. S., Cruz-Pimentel, M., Patil, N. S., Popovic, M. M., Pandya, B. U., Shor, R., Pereira, A., & Muni, R. H. (2024) · Eye · doi:10.1038/s41433-024-03074-5 · Cited by 7 · see citing works
Self-citationMihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2024) · Journal of Cataract & Refractive Surgery · doi:10.1097/j.jcrs.0000000000001317 · Cited by 7 · see citing works
Self-citationEvaluating the Performance of Large Language Models on a Neurology Board-Style ExaminationPreprint
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · medRxiv · doi:10.1101/2023.07.13.23292598 · Cited by 7 · see citing works
Hamzeh, N., Lidder, A. K., Feder, R. S., Sarmiento, E. A., Mirza, R. G., Thau, A. J., & Tanna, A. P. (2025) · Ophthalmology Science · doi:10.1016/j.xops.2025.101007 · Cited by 6 · see citing works
Exploring the Test-Taking Capabilities of Chatbots—From Surgeon to Sommelier
Chia, M. A., & Keane, P. A. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.3003 · Cited by 6 · see citing works
Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics
Koga, S., & Du, W. D. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2303 · Cited by 5 · see citing works
Sabaner, M. C., & Yozgat, Z. (2025) · Turkish Journal of Ophthalmology · doi:10.4274/tjo.galenos.2025.27895 · Cited by 5 · see citing works
The application of artificial intelligence-generated content in ophthalmology education
Wang, Y., Zhao, Y., & Li, J. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1617537 · Cited by 4 · see citing works
Advances in Artificial Intelligence Chatbot Technology in Ophthalmology—Reply
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.4623 · Cited by 4 · see citing works
Self-citationNeed for Custom Artificial Intelligence Chatbots in Ophthalmology
Mihalache, A., Popovic, M. M., & Muni, R. H. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2738 · Cited by 4 · see citing works
Self-citationFundus photograph interpretation of common retinal disorders by artificial intelligence chatbots
Mihalache, A., Huang, R. S., Popovic, M. M., Yan, P., Muni, R. H., Huang, S. S., & Wong, D. T. (2025) · AJO International · doi:10.1016/j.ajoint.2025.100154 · Cited by 3 · see citing works
Self-citationWu, K. Y., Qian, S. Y., & Marchand, M. (2025) · Diagnostics · doi:10.3390/diagnostics15141820 · Cited by 3 · see citing works
Audit-style framework for evaluating bias in large language models
Baldwin, P. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1592037 · Cited by 6 · see citing works
Ming, S., Guo, Q., Cheng, W., & Lei, B. (2023) · doi:10.2196/preprints.52784 · Cited by 2 · see citing works
Yu, H., Fu, Y., Jiang, B., Fan, P., Shen, L., Gao, L., Lu, H., Zhang, Y., Dong, S., Lv, J., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-3644163/v1 · Cited by 2 · see citing works
Chen, J. L., Lu, A. J., Verma, R., Wang, L., Koch, D. D., & Chen, A. J. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101130 · Cited by 1 · see citing works
Kaya, N., Şengül, F., Çelikel, P., & Erdoğan, A. Ş. (2026) · European Journal Of Dental Education · doi:10.1111/eje.70295 · Cited by 1 · see citing works
Zhou, B., Sun, L., Xia, H.-S., Tan, B., Shi, Y.-L., Liu, H., Qin, C., Wen, C., Wang, L., Lu, D., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5116563 · Cited by 1 · see citing works
JAMA Ophthalmology—The Year in Review, 2023
Bressler, N. M. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.0435
Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics—Reply
Mihalache, A., Popovic, M. M., & Muni, R. H. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2306
Self-citationLin, C., Chen, C.-Y., Wu, P.-L., Chen, C., & Lai, C.-H. (2025) · Taiwan Journal of Ophthalmology · doi:10.4103/tjo.tjo-d-24-00164
Enhancing Educational Feedback—A Paradigm Shift Through AI-Driven Chatbots
Ghosh, S., Sharma, S., & Mahata, S. K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3287-9_34
Lindsay, C. V., Shenoy, D. A., Martin, A. N., Clipper, C., Shah, K. N., Lidsky, M. E., Nussbaum, D. P., & D, R. S. M. (2025) · doi:10.2196/preprints.81374
Trends and advances in ChatGPT applications in ophthalmology
Zheng, H., Dong, H., & Zhao, H. (2025) · Journal Français d Ophtalmologie · doi:10.1016/j.jfo.2025.104622
Transforming Ophthalmic Education With Large Language Models
Mihalache, A., Popovic, M. M., & Muni, R. H. (2025) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2025.3759
Self-citationLindsay, C. V., Shenoy, D. A., Martin, A. N., Clipper, C., Shah, K. N., Lidsky, M. E., Nussbaum, D. P., & D, R. S. M. (2026) · JMIR Perioperative Medicine · doi:10.2196/81374
Fei, F., 叶璐, Rao, J., Liang, Z., Yang, J., Zou, Y., & Jiang, L. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1871740 · Cited by 1 · see citing works
Artificial Intelligence Chatbot–Driven Detection of Keratoconus From Corneal Tomography
Mihalache, A., Huang, R. S., Nichani, P. A. H., Popovic, M. M., & Chan, C. C. (2026) · Cornea · doi:10.1097/ico.0000000000004279
Self-citationTabuchi, H., Oshika, T., Akada, M., Toba, Y., & Gomi, F. (2026) · AJO International · doi:10.1016/j.ajoint.2026.100328
Performance of Frontier Multimodal Large Language Models on Ophthalmology Cases
Kaleem, S., Mihalache, A., Peng, H., Rahman, U., Timajchi, E., Seralathan, K., Kaleem, S., Popovic, M. M., & Muni, R. H. (2026) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2026.4512
Self-citation
- Authors
- Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H.
- Affiliations
- Western University (CA)
- University of Toronto (CA)
- St. Michael's Hospital (CA)
- Corresponding author
- Muni, Rajeev H.
- Venue
- JAMA Ophthalmology
- Publisher
- American Medical Association (AMA)
- Published
- 2023-08-01
- Volume
- 141
- Issue
- 8
- Pages
- 798
- ISSN
- 2168-6165 (ISSN-L)
- PMC
- PMC10346493
- OpenAlex
- W4384120659
- Cited by
- 67
- References
- 3
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 31 · 2023: 12
- FWCI
- 2.68
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- JAMA Ophthalmology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- chatbots, artificial intelligence, ophthalmology, board certification examination, exam preparation, answer accuracy, knowledge assessment
- MeSH terms
- Artificial Intelligence, Eye, Humans, Test Taking Skills, Surgeons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
