The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Open accessAbstract
Importance: ChatGPT is an artificial intelligence (AI) chatbot that has significant societal implications. Training curricula using AI are being developed in medicine, and the performance of chatbots in ophthalmology has not been characterized. Objective: To assess the performance of ChatGPT in answering practice questions for board certification in ophthalmology. Design, Setting, and Participants: This cross-sectional study used a consecutive sample of text-based multiple-choice questions provided by the OphthoQuestions practice question bank for board certification examination preparation. Of 166 available multiple-choice questions, 125 (75%) were text-based. Exposures: ChatGPT answered questions from January 9 to 16, 2023, and on February 17, 2023. Main Outcomes and Measures: Our primary outcome was the number of board certification examination practice questions that ChatGPT answered correctly. Our secondary outcomes were the proportion of questions for which ChatGPT provided additional explanations, the mean length of questions and responses provided by ChatGPT, the performance of ChatGPT in answering questions without multiple-choice options, and changes in performance over time. Results: In January 2023, ChatGPT correctly answered 58 of 125 questions (46%). ChatGPT’s performance was the best in the category general medicine (11/14; 79%) and poorest in retina and vitreous (0%). The proportion of questions for which ChatGPT provided additional explanations was similar between questions answered correctly and incorrectly (difference, 5.82%; 95% CI, −11.0% to 22.0%; χ 21 = 0.45; P = .51). The mean length of questions was similar between questions answered correctly and incorrectly (difference, 21.4 characters; SE, 36.8; 95% CI, −51.4 to 94.3; t = 0.58; df = 123; P = .22). The mean length of responses was similar between questions answered correctly and incorrectly (difference, −80.0 characters; SE, 65.4; 95% CI, −209.5 to 49.5; t = −1.22; df = 123; P = .22). ChatGPT selected the same multiple-choice response as the most common answer provided by ophthalmology trainees on OphthoQuestions 44% of the time. In February 2023, ChatGPT provided a correct response to 73 of 125 multiple-choice questions (58%) and 42 of 78 stand-alone questions (54%) without multiple-choice options. Conclusions and Relevance: ChatGPT answered approximately half of questions correctly in the OphthoQuestions free trial for ophthalmic board certification preparation. Medical professionals and trainees should appreciate the advances of AI in medicine while acknowledging that ChatGPT as used in this investigation did not answer sufficient multiple-choice questions correctly for it to provide substantial assistance in preparing for board certification at this time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kirkwood, B. R., & Sterne, J. A. C. (2014) · Bristol Research (University of Bristol) · Cited by 2,820
Smithson, M. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781446218525 · Cited by 1,871
Higgins (2002) · doi:10.1002/(issn)1097-0258 · Cited by 1,248
Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations
Campbell, I. (2007) · Statistics in Medicine · doi:10.1002/sim.2832 · Cited by 912
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
OpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare
Aydın, Ö., & Karaarslan, E. (2022) · doi:10.2139/ssrn.4308687 · Cited by 391
The analysis of 2 × 2 contingency tables—Yet again
Richardson, J. T. E. (2011) · Statistics in Medicine · doi:10.1002/sim.4116 · Cited by 290
Cited by
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C.-H., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 310
Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154
Moshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124
Performance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 172
Metadata
- Authors
- Mihalache, A., Popovic, M. M., & Muni, R. H.
- Venue
- JAMA Ophthalmology
- Publisher
- American Medical Association (AMA)
- Published
- 2023-06-01
- Volume
- 141
- Issue
- 6
- Pages
- 589
- ISSN
- 2168-6165 (ISSN-L)
- OpenAlex
- W4367175039
- Cited by
- 304
- References
- 15
- FWCI
- 10.45
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. https://app.cochat.ai/papers/10.1001/jamaophthalmol.2023.1144
- The paper
- Mihalache, A., Popovic, M. M., & Muni, R. H. (2023). Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmology. https://doi.org/10.1001/jamaophthalmol.2023.1144
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.