Conversational AI Models for ophthalmic diagnosis: Comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
With the rapidly growing field of conversational artificial intelligence (AI), it is becoming increasingly likely that these technologies will revolutionize the way physicians diagnose and treat patients. The purpose of this study is to evaluate the use of conversational AI language models, specifically ChatGPT, for the diagnosis of ophthalmic disease, and to compare it to existing tools, namely the Isabel Pro Differential Diagnosis Generator. Prospective, comparative evaluation of ChatGPT and Isabel in formulating provisional and differential diagnoses from a set of rich text case report descriptions. Ten ophthalmology patient cases were selected at random from a publicly available online database of ophthalmic case reports. The text details of each case were input into ChatGPT and Isabel. Their ability to identify the actual diagnosis and provide relevant differential diagnoses was compared. ChatGPT identified the correct diagnosis in 9/10 cases while having the correct diagnosis listed in all 10/10 of its lists of differentials. Isabel identified only 1/10 provisional diagnoses correctly, however it included the correct diagnosis in 7/10 of its differential diagnosis lists. The median position of the correct diagnosis in the ranked differential lists was 1.0 (IQR 1.0 to 2.8) for ChatGPT versus 5.5 (IQR 3.3 to 10.0) for Isabel. Conversational AI models such as ChatGPT have potential value in the diagnosis of ophthalmic conditions, particularly for primary-care providers. As further iterations of models are deployed, additional studies investigating their capabilities are needed to determine the best ways to integrate them into practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Checklists to Reduce Diagnostic Errors
Ely, J. W., Graber, M. L., & Croskerry, P. (2011) · Academic Medicine · doi:10.1097/acm.0b013e31820824cd · Cited by 366
Cited by
ChatGPT for healthcare services: An emerging stage for an innovative perspective
Javaid, M., Haleem, A., & Singh, R. P. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100105 · Cited by 429
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311
“Deep learning has been applied to promising effect for generating DDx in a number of specialties including radiology 4 , ophthalmology 5 and dermatology 3 , but such systems lack the interactive capabilities to fluently assist a user through communication in natural language.”
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189
Metadata
- Authors
- Balas, M., & Ing, E. B.
- Venue
- JFO Open Ophthalmology
- Publisher
- Elsevier BV
- Published
- 2023-03
- Volume
- 1
- Pages
- 100005
- ISSN
- 2949-8899 (ISSN-L)
- OpenAlex
- W4323030608
- Cited by
- 110
- References
- 7
- FWCI
- 3.81
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Conversational AI Models for ophthalmic diagnosis: Comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator. https://app.cochat.ai/papers/10.1016/j.jfop.2023.100005
- The paper
- Balas, M., & Ing, E. B. (2023). Conversational AI Models for ophthalmic diagnosis: Comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator. JFO Open Ophthalmology. https://doi.org/10.1016/j.jfop.2023.100005
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.