The DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index, and 1 are not checked yet.
Evaluation and Comparison of Ophthalmic Scientific Abstracts and References by Current Artificial Intelligence Chatbots
Open accessAbstract
Importance: Language-learning model–based artificial intelligence (AI) chatbots are growing in popularity and have significant implications for both patient education and academia. Drawbacks of using AI chatbots in generating scientific abstracts and reference lists, including inaccurate content coming from hallucinations (ie, AI-generated output that deviates from its training data), have not been fully explored. Objective: To evaluate and compare the quality of ophthalmic scientific abstracts and references generated by earlier and updated versions of a popular AI chatbot. Design, Setting, and Participants: This cross-sectional comparative study used 2 versions of an AI chatbot to generate scientific abstracts and 10 references for clinical research questions across 7 ophthalmology subspecialties. The abstracts were graded by 2 authors using modified DISCERN criteria and performance evaluation scores. Main Outcome and Measures: Scores for the chatbot-generated abstracts were compared using the t test. Abstracts were also evaluated by 2 AI output detectors. A hallucination rate for unverifiable references generated by the earlier and updated versions of the chatbot was calculated and compared. Results: The mean modified AI-DISCERN scores for the chatbot-generated abstracts were 35.9 and 38.1 (maximum of 50) for the earlier and updated versions, respectively ( P = .30). Using the 2 AI output detectors, the mean fake scores (with a score of 100% meaning generated by AI) for the earlier and updated chatbot-generated abstracts were 65.4% and 10.8%, respectively ( P = .01), for one detector and were 69.5% and 42.7% ( P = .17) for the second detector. The mean hallucination rates for nonverifiable references generated by the earlier and updated versions were 33% and 29% ( P = .74). Conclusions and Relevance: Both versions of the chatbot generated average-quality abstracts. There was a high hallucination rate of generating fake references, and caution should be used when using these AI resources for health education or academic purposes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings align with those of Hua et al [25] who evaluated hallucinations in AI-generated ophthalmic scientific abstracts and references.”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings showed variations in the RHS across different AI chatbots, ranging from almost null for SciSpace and Elicit to a critically high degree of hallucination for ChatGPT [25].”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [25]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437
A Randomized, Controlled Trial of Corticosteroids in the Treatment of Acute Optic Neuritis
Beck, R. W., Cleary, P. A., Anderson, M. M., Keltner, J. L., Shults, W. T., Kaufman, D. I., Buckley, E. G., Corbett, J. J., Kupersmith, M. J., Miller, N. R., et al. (1992) · New England Journal of Medicine · doi:10.1056/nejm199202273260901 · Cited by 1,349
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Tools such as ChatGPT threaten transparent science; here are our ground rules for their use
(2023) · Nature · doi:10.1038/d41586-023-00191-1 · Cited by 605
Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?
O’Connor, S., & ChatGPT (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103537 · Cited by 585
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265
Research works printed but not linked: 2
- Sage. ChatGPT and generative AI. Accessed May 18, 2023. https://us.sagepub.com/en-us/nam/chatgpt-and-generative-ai
- Elsevier. Publishing ethics—the use of generative AI and AI-assisted technologies in scientific writing. Accessed May 18, 2023. https://www.elsevier.com/about/policies/publishing-ethics
Cited by
Sorted by citations
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024) · JMIR Medical Informatics · doi:10.2196/54345 · Cited by 211
“Our findings showed variations in the RHS across different AI chatbots, ranging from almost null for SciSpace and Elicit to a critically high degree of hallucination for ChatGPT [25].”
BackgroundResultRevolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine
Zhang, K., Meng, X., Yan, X., Ji, J., Liu, J., Xu, H., Zhang, H., Liu, D., Wang, J., Wang, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59069 · Cited by 148
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146
Democratizing Financial Knowledge with ChatGPT by OpenAI: Unleashing the Power of TechnologyPreprint
Yue, T., Au, D., Au, C. C., & Iu, K. Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4346152 · Cited by 74
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 61
Metadata
- Authors
- Hua, H.-U., Kaakour, A.-H., Rachitskaya, A., Srivastava, S., Sharma, S., & Mammo, D. A.
- Affiliations
- Cleveland Clinic (US)
- Corresponding author
- Mammo, Danny A.
- ORCID
- Venue
- JAMA Ophthalmology
- Publisher
- American Medical Association (AMA)
- Published
- 2023-09-01
- Volume
- 141
- Issue
- 9
- Pages
- 819
- ISSN
- 2168-6165 (ISSN-L)
- PMC
- PMC10375387
- OpenAlex
- W4385294579
- Cited by
- 76
- References
- 14
- Citations by year
- 2026: 16 · 2025: 28 · 2024: 26 · 2023: 6
- FWCI
- 3.03
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- JAMA Ophthalmology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- AI chatbots, large language models, ophthalmology, hallucinations, DISCERN criteria, AI-generated content quality, ophthalmology subspecialties
- MeSH terms
- Artificial Intelligence, Cross-Sectional Studies, Eye, Hallucinations, Health Education, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation and Comparison of Ophthalmic Scientific Abstracts and References by Current Artificial Intelligence Chatbots. https://app.cochat.ai/papers/10.1001/jamaophthalmol.2023.3119
- The paper
- Hua, H.-U., Kaakour, A.-H., Rachitskaya, A., Srivastava, S., Sharma, S., & Mammo, D. A. (2023). Evaluation and Comparison of Ophthalmic Scientific Abstracts and References by Current Artificial Intelligence Chatbots. JAMA Ophthalmology. https://doi.org/10.1001/jamaophthalmol.2023.3119
