Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Large language models (LLMs) are garnering wide interest due to their human-like and contextually relevant responses. However, LLMs' accuracy across specific medical domains has yet been thoroughly evaluated. Myopia is a frequent topic which patients and parents commonly seek information online. Our study evaluated the performance of three LLMs namely ChatGPT-3.5, ChatGPT-4.0, and Google Bard, in delivering accurate responses to common myopia-related queries. METHODS: We curated thirty-one commonly asked myopia care-related questions, which were categorised into six domains-pathogenesis, risk factors, clinical presentation, diagnosis, treatment and prevention, and prognosis. Each question was posed to the LLMs, and their responses were independently graded by three consultant-level paediatric ophthalmologists on a three-point accuracy scale (poor, borderline, good). A majority consensus approach was used to determine the final rating for each response. 'Good' rated responses were further evaluated for comprehensiveness on a five-point scale. Conversely, 'poor' rated responses were further prompted for self-correction and then re-evaluated for accuracy. FINDINGS: ChatGPT-4.0 demonstrated superior accuracy, with 80.6% of responses rated as 'good', compared to 61.3% in ChatGPT-3.5 and 54.8% in Google Bard (Pearson's chi-squared test, all p ≤ 0.009). All three LLM-Chatbots showed high mean comprehensiveness scores (Google Bard: 4.35; ChatGPT-4.0: 4.23; ChatGPT-3.5: 4.11, out of a maximum score of 5). All LLM-Chatbots also demonstrated substantial self-correction capabilities: 66.7% (2 in 3) of ChatGPT-4.0's, 40% (2 in 5) of ChatGPT-3.5's, and 60% (3 in 5) of Google Bard's responses improved after self-correction. The LLM-Chatbots performed consistently across domains, except for 'treatment and prevention'. However, ChatGPT-4.0 still performed superiorly in this domain, receiving 70% 'good' ratings, compared to 40% in ChatGPT-3.5 and 45% in Google Bard (Pearson's chi-squared test, all p ≤ 0.001). INTERPRETATION: Our findings underscore the potential of LLMs, particularly ChatGPT-4.0, for delivering accurate and comprehensive responses to myopia-related queries. Continuous strategies and evaluations to improve LLMs' accuracy remain crucial. FUNDING: Dr Yih-Chung Tham was supported by the National Medical Research Council of Singapore (NMRC/MOH/HCSAINV21nov-0001).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 662
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
Wu, P.-C., Huang, H.-M., Yu, H.-J., Fang, P.-C., & Chen, C.-T. (2016) · Asia-Pacific Journal of Ophthalmology · doi:10.1097/apo.0000000000000236 · Cited by 357
Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J. S., Shin, J. H., Zadnik Sullivan, P. L., Cielo, D., Oyelese, A. A., Doberstein, C. E., Telfeian, A. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002551 · Cited by 354
AI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346
Large language models and the perils of their hallucinations
Azamfirei, R., Kudchadkar, S. R., & Fackler, J. (2023) · Critical Care · doi:10.1186/s13054-023-04393-x · Cited by 323
Bao, J., Huang, Y., Li, X., Yang, A., Zhou, F., Wu, J., Wang, C., Li, Y., Lim, E. W., Spiegel, D. P., et al. (2022) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2022.0401 · Cited by 304
Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment
Mihalache, A., Popovic, M. M., & Muni, R. H. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.1144 · Cited by 304
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289
Ethics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282
Will ChatGPT transform healthcare?
(2023) · Nature Medicine · doi:10.1038/s41591-023-02289-5 · Cited by 214
Xie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198
Machine Learning in Healthcare Communication
Siddique, S., & Chow, J. C. L. (2021) · Encyclopedia · doi:10.3390/encyclopedia1010021 · Cited by 190
Impact of ChatGPT on medical chatbots as a disruptive technology
Chow, J. C. L., Sanders, L., & Li, K. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1166014 · Cited by 188
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K., Mansour, H. A., Abishek, R. M., Xu, D., Sridhar, J., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 178
ChatGPT and Ophthalmology: Exploring Its Potential with Discharge Summaries and Operative Notes
Singh, S., Djalilian, A. R., & Ali, M. J. (2023) · Seminars in Ophthalmology · doi:10.1080/08820538.2023.2209166 · Cited by 169
The credibility of dietary advice formulated by ChatGPT: Robo-diets for people with food allergies
Niszczota, P., & Rybicka, I. (2023) · Nutrition · doi:10.1016/j.nut.2023.112076 · Cited by 163
Artificial intelligence‐based chatbot patient information on common retinal diseases using ChatGPT
Potapenko, I. O., Boberg‐Ans, L. C., Hansen, M. S., Klefter, O. N., van Dijk, E. H. C., & Subhi, Y. (2023) · Acta Ophthalmologica · doi:10.1111/aos.15661 · Cited by 148
Prevention of Myopia Onset with 0.025% Atropine in Premyopic Children
Fang, P., Chung, M., Yu, H.-J., & Wu, P. (2010) · Journal of Ocular Pharmacology and Therapeutics · doi:10.1089/jop.2009.0135 · Cited by 127
Evaluating the use of large language model in identifying top research questions in gastroenterology
Lahat, A., Shachar, E., Avidan, B., Shatz, Z., Glicksberg, B. S., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-31412-2 · Cited by 111
Lahat, A., Shachar, E., Avidan, B., Glicksberg, B. S., & Klang, E. (2023) · Diagnostics · doi:10.3390/diagnostics13111950 · Cited by 108
More Than One-Third of US Individuals Use the Internet to Self-diagnose
Kuehn, B. M. (2013) · JAMA · doi:10.1001/jama.2013.629 · Cited by 91
Raimondi, R., Tzoumas, N., Salisbury, T., Simplicio, S. D., Romano, M. R., Bommireddy, T., Chawla, H., Chen, Y., Connolly, S., Omda, S. E., et al. (2023) · Eye · doi:10.1038/s41433-023-02563-3 · Cited by 87
Artificial intelligence-based text generators in hepatology: ChatGPT is just the beginning
Ge, J., & Lai, J. C. (2023) · Hepatology Communications · doi:10.1097/hc9.0000000000000097 · Cited by 83
ChatGPT and Lacrimal Drainage Disorders: Performance and Scope of Improvement
Ali, M. J. (2023) · Ophthalmic Plastic and Reconstructive Surgery · doi:10.1097/iop.0000000000002418 · Cited by 82
Rasmussen, M. L. R., Larsen, A., Subhi, Y., & Potapenko, I. O. (2023) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-023-06078-1 · Cited by 76
Carlà, M. M., Boselli, F., Giannuzzi, F., Gambini, G., Caporossi, T., Vico, U. D., Savastano, A., Baldascino, A., Rizzo, C., Kilian, R. A., et al. (2022) · Vision · doi:10.3390/vision6020020 · Cited by 40
GPT-4 vs. GPT-3.5: A Concise Showdown
Koubaa, A. (2023) · doi:10.20944/preprints202303.0422.v1 · Cited by 37
Is Dietary Vitamin A Associated with Myopia from Adolescence to Young Adulthood?
Ng, F. J., Mackey, D. A., O’Sullivan, T. A., Oddy, W. H., & Yazar, S. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.6.29 · Cited by 33
ChatGPT as a tool for conducting literature review for dry eye disease
Singh, S., & Watson, S. L. (2023) · Clinical and Experimental Ophthalmology · doi:10.1111/ceo.14268 · Cited by 20
Lahat, A., & Klang, E. (2023) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x231155520 · Cited by 16
Cited by
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
“38), the latter of which has been regarded as state of the art on general NLP tasks 21 and has demonstrated strong performance on biomedical NLP tasks, such as medical examinations 65–67 .”
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01029-4 · Cited by 390
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Moulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Luo, X., Tham, Y. C., Giuffrè, M., Ranisch, R., Daher, M., Lam, K., Eriksen, A. V., Hsu, C.-W., Ozaki, A., Moraes, F. Y. D., et al. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2025-113825 · Cited by 97
Self-citation
Metadata
- Authors
- Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C.-H., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., Cheng, C.-Y., Koh, V. T. C., & Tham, Y.-C.
- Venue
- eBioMedicine
- Publisher
- Elsevier BV
- Published
- 2023-09
- Volume
- 95
- Pages
- 104770
- ISSN
- 2352-3964 (ISSN-L)
- OpenAlex
- W4386110374
- Cited by
- 310
- References
- 43
- FWCI
- 10.73
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.