Artificial intelligence chatbot performance in triage of ophthalmic conditions
Open accessPublished version of a preprint
Abstract
Background Timely access to human expertise for affordable and efficient triage of ophthalmic conditions is inconsistent. With recent advancements in publicly available artificial intelligence (AI) chatbots, the lay public may turn to these tools for triage of ophthalmic complaints. Validation studies are necessary to evaluate the performance of AI chatbots as triage tools and inform the public regarding their safety. Objective To evaluate the triage performance of AI chatbots for ophthalmic conditions. Design Cross-sectional study. Setting Single centre. Participants Ophthalmology trainees, OpenAI ChatGPT (GPT-4), Bing Chat, and WebMD Symptom Checker. Methods Forty-four clinical vignettes representing common ophthalmic complaints were developed, and a standardized pathway of prompts was presented to each tool in March 2023. Primary outcomes were proportion of responses with the correct diagnosis listed in the top 3 possible diagnoses and proportion with correct triage urgency. Ancillary outcomes included presence of grossly inaccurate statements, mean reading grade level, mean response word count, proportion with attribution, and most common sources cited. Results The ophthalmologists in training, ChatGPT, Bing Chat, and the WebMD Symptom Checker listed the appropriate diagnosis among the top 3 suggestions in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases, respectively. Triage urgency was appropriate in 38 (86%), 43 (98%), and 37 (84%) cases for ophthalmology trainees, ChatGPT, and Bing Chat, correspondingly. Conclusions ChatGPT using the GPT-4 model offered high diagnostic and triage accuracy that was comparable with that of ophthalmology trainees with no grossly inaccurate statements. Bing Chat had lower accuracy and a tendency to overestimate triage urgency.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Lyons ophthalmic set was replaced by the registered Zandi set, because its vignette text is published only as an image.
“The Lyons ophthalmic set [16] was replaced by the registered Zandi set, because its vignette text is published only as an image.”
From Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models (2026) · cites this paper as [16]
The Clinical vignette design section confirms that Lyons and colleagues developed an ophthalmic vignette set, so the citation identifies the set correctly. Our copy does not establish that its vignette text was published only as an image. The replacement with the Zandi set is the citing authors’ decision, not a finding attributed to Lyons. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Using AI to write scholarly publications
Hosseini, M., Rasmussen, L. M., & Resnik, D. B. (2024) · Accountability in Research · doi:10.1080/08989621.2023.2168535 · Cited by 286 · see citing works
Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- Biswas SS. Role of ChatGPT in public health. Ann Biomed Eng 2023;51:868-9.
- Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical edu- cation using large language models. PLOS Digit Health 2023;2:e0000198.
- Rahimi F, Talebi Bezmin Abadi A. ChatGPT and publication ethics. Arch Med Res 2023;54:272-4.
- Thapa DK, Visentin DC, Kornhaber R, West S, Cleary M. The influence of online health information on health decisions: a systematic review. Patient Educ Couns 2021; 104:770-84.
- Hall LN, Jeng-Miller KW, Gardiner M, Kim EL. Utilization trends of an ophthalmology-specific emergency department: the Massachusetts Eye and Ear experience. Digit J Ophthal- mol 2021;26:31-5.
- Channa R, Zafar SN, Canner JK, Haring RS, Schneider EB, Friedman DS. Epidemiology of eye-related emergency depart- ment visits. JAMA Ophthalmol 2016;134:312-9.
- Lidal IB, Holte HH, Vist GE. Triage systems for pre-hospital emergency medical services-a systematic review. Scand. J Trauma Resusc Emerg Med 2013;21:28.
- Linzer M, Jin JO, Shah P, et al. Trends in clinician burnout with associated mitigating and aggravating factors during the COVID-19 pandemic. JAMA Health Forum 2022;3: e224163.
- Balas M, Ing E. Conversational AI models for ophthalmic diagnosis: comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator. JFO Open Ophthalmol 2023;1:100005. doi: 10.1016/j.jfop.2023. 100005. · DOI
- Primativo S, Spinelli D, Zoccolotti P, De Luca M, Martelli M. Perceptual and cognitive factors imposing "speed limits" on reading rate: a study with the rapid serial visual presentation. PloS One 2016;11:e0153786.
- Kloosterboer A, Yannuzzi NA, Patel NA, Kuriyan AE, Srid- har J. Assessment of the quality, content, and readability of freely available online information for patients regarding dia- betic retinopathy. JAMA Ophthalmol 2019;137:1240-5.
- Partin S, Westfall E, Sanda G, et al. Readability, content, and accountability assessment of online health information for ret- initis pigmentosa and retinitis pigmentosa treatment options. Ophthalmic Genet 2023;44:43-8.
- Flanagin A, Bibbins-Domingo K, Berkwits M, Christiansen SL. Nonhuman "authors" and implications for the integrity of scientific publication and medical knowledge. JAMA 2023; 329:637-9.
Cited by
Sorted by citations
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203 · see citing works
Mixed methods assessment of the influence of demographics on medical advice of ChatGPT
Andreadis, K., Newman, D. R., Twan, C., Shunk, A., Mann, D. M., & Stevens, E. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae086 · Cited by 20 · see citing works
Deobhakta, A., Chang, J., Chen, R. W. S., Jaipalli, S., Pinhas, A., Tania Tai, T. Y., & Rosen, R. B. (2026) · doi:10.64898/2026.10.09.26365153
- Authors
- Lyons, R. J., Arepalli, S. R., Fromal, O., Choi, J. D., & Jain, N.
- Affiliations
- Emory University (US)
- Corresponding author
- Jain, Nieraj
- Venue
- Canadian Journal of Ophthalmology
- Publisher
- Elsevier BV
- Published
- 2024-08
- Volume
- 59
- Issue
- 4
- Pages
- e301-e308
- ISSN
- 0008-4182 (ISSN-L)
- OpenAlex
- W4385707210
- Cited by
- 101
- References
- 3
- Citations by year
- 2026: 22 · 2025: 35 · 2024: 38 · 2023: 5
- FWCI
- 3.71
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Canadian Journal of Ophthalmology (publishedVersion)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- Research to Prevent Blindness
- Foundation Fighting Blindness
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- AI chatbots, ChatGPT, Bing Chat, diagnostic accuracy, triage urgency, clinical vignettes, ophthalmology trainees, large language models
- MeSH terms
- Ophthalmologists, Adult, Artificial Intelligence, Cross-Sectional Studies, Eye Diseases, Female, Humans, Male, Ophthalmology, Triage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
