Cited by 80Artificial Intelligence in Healthcare and EducationAcademic Writing and PublishingAcademic integrity and plagiarism

The DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index, and 1 are not checked yet.

Evaluation and Comparison of Ophthalmic Scientific Abstracts and References by Current Artificial Intelligence Chatbots

Open access

Abstract

Importance: Language-learning model–based artificial intelligence (AI) chatbots are growing in popularity and have significant implications for both patient education and academia. Drawbacks of using AI chatbots in generating scientific abstracts and reference lists, including inaccurate content coming from hallucinations (ie, AI-generated output that deviates from its training data), have not been fully explored. Objective: To evaluate and compare the quality of ophthalmic scientific abstracts and references generated by earlier and updated versions of a popular AI chatbot. Design, Setting, and Participants: This cross-sectional comparative study used 2 versions of an AI chatbot to generate scientific abstracts and 10 references for clinical research questions across 7 ophthalmology subspecialties. The abstracts were graded by 2 authors using modified DISCERN criteria and performance evaluation scores. Main Outcome and Measures: Scores for the chatbot-generated abstracts were compared using the t test. Abstracts were also evaluated by 2 AI output detectors. A hallucination rate for unverifiable references generated by the earlier and updated versions of the chatbot was calculated and compared. Results: The mean modified AI-DISCERN scores for the chatbot-generated abstracts were 35.9 and 38.1 (maximum of 50) for the earlier and updated versions, respectively ( P = .30). Using the 2 AI output detectors, the mean fake scores (with a score of 100% meaning generated by AI) for the earlier and updated chatbot-generated abstracts were 65.4% and 10.8%, respectively ( P = .01), for one detector and were 69.5% and 42.7% ( P = .17) for the second detector. The mean hallucination rates for nonverifiable references generated by the earlier and updated versions were 33% and 29% ( P = .74). Conclusions and Relevance: Both versions of the chatbot generated average-quality abstracts. There was a high hallucination rate of generating fake references, and caution should be used when using these AI resources for health education or academic purposes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamaophthalmol.2023.3119

Cites

14 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 14 references

Research works printed but not linked: 2

  1. Sage. ChatGPT and generative AI. Accessed May 18, 2023. https://us.sagepub.com/en-us/nam/chatgpt-and-generative-ai
  2. Elsevier. Publishing ethics—the use of generative AI and AI-assisted technologies in scientific writing. Accessed May 18, 2023. https://www.elsevier.com/about/policies/publishing-ethics

Cited by

Sorted by citations

Metadata

Authors
Hua, H.-U., Kaakour, A.-H., Rachitskaya, A., Srivastava, S., Sharma, S., & Mammo, D. A.
Affiliations
Cleveland Clinic (US)
Corresponding author
Mammo, Danny A.
Venue
JAMA Ophthalmology
Published
2023-09-01
Volume
141
Issue
9
Pages
819
ISSN
2168-6165 (ISSN-L)
OpenAlex
W4385294579
Cited by
76
References
14
Citations by year
2026: 16 · 2025: 28 · 2024: 26 · 2023: 6
FWCI
3.03
Type
journal article
Language
en
Open access
Locations
  • JAMA Ophthalmology (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
AI chatbots, large language models, ophthalmology, hallucinations, DISCERN criteria, AI-generated content quality, ophthalmology subspecialties
MeSH terms
Artificial Intelligence, Cross-Sectional Studies, Eye, Hallucinations, Health Education, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review