CoChat
Sign Up
Cited by 97Artificial Intelligence in Healthcare and EducationHealth Literacy and Information AccessibilitySocial Media in Health Education

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis

Open access
Ghanem, Yazid K., Rouhi, Armaun D., Al-Houssan, Ammr, Saleh, Zena, Moccia, Matthew C., Joshi, Hansa, Dumon, Kristoffel R., Hong, Young +3 more·2024·Surgical Endoscopy
DOI 10.1007/s00464-024-10739-5PMC PMC11078845
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: Generative artificial intelligence (AI) chatbots have recently been posited as potential sources of online medical information for patients making medical decisions. Existing online patient-oriented medical information has repeatedly been shown to be of variable quality and difficult readability. Therefore, we sought to evaluate the content and quality of AI-generated medical information on acute appendicitis. Methods: A modified DISCERN assessment tool, comprising 16 distinct criteria each scored on a 5-point Likert scale (score range 16–80), was used to assess AI-generated content. Readability was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. Four popular chatbots, ChatGPT-3.5 and ChatGPT-4, Bard, and Claude-2, were prompted to generate medical information about appendicitis. Three investigators independently scored the generated texts blinded to the identity of the AI platforms. Results: ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 had overall mean (SD) quality scores of 60.7 (1.2), 62.0 (1.0), 62.3 (1.2), and 51.3 (2.3), respectively, on a scale of 16–80. Inter-rater reliability was 0.81, 0.75, 0.81, and 0.72, respectively, indicating substantial agreement. Claude-2 demonstrated a significantly lower mean quality score compared to ChatGPT-4 (p = 0.001), ChatGPT-3.5 (p = 0.005), and Bard (p = 0.001). Bard was the only AI platform that listed verifiable sources, while Claude-2 provided fabricated sources. All chatbots except for Claude-2 advised readers to consult a physician if experiencing symptoms. Regarding readability, FKGL and FRE scores of ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 were 14.6 and 23.8, 11.9 and 33.9, 8.6 and 52.8, 11.0 and 36.6, respectively, indicating difficulty readability at a college reading skill level. Conclusion: AI-generated medical information on appendicitis scored favorably upon quality assessment, but most either fabricated sources or did not provide any altogether. Additionally, overall readability far exceeded recommended levels for the public. Generative AI platforms demonstrate measured potential for patient education and engagement about appendicitis.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00464-024-10739-5

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some investigators assessed AI-generated medical information, including the ability to verify references and publication date, but did not pay attention specifically to certain identifiers of references that help verify their authenticity.

    “Some investigators assessed the AI-generated medical information, including the ability to verify references and its publication date, but did not pay attention specifically to certain identifiers of references that help to verify their authenticity [18].”

    From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [18]

    The Materials and methods section says the study assessed AI-generated appendicitis information and whether its sources could be verified. Our copy does not show that it assessed publication dates. The Discussion explicitly mentions fabricated Digital Object Identifier (DOI) numbers, so the claim that the investigators overlooked reference identifiers is not established by the cited paper. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. [2]Shah NH, Entwistle DA, Pfeffer M (2023) Creation and adoption of large language models in medicine. JAMA 330(9):866. 10.1001/jama.2023.14217 37548965 10.1001/jama.2023.14217 · DOI · Paper page
  2. [5]Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Online resources for patients considering hiatal hernia repair: a quality and readability analysis. J Gastrointest Surg 27(3):598–600. 10.1007/s11605-022-05460-4 36127551 10.1007/s11605-022-05460-4 · DOI
  3. [6]Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Quality and readability of online patient information on adolescent bariatric surgery. Obes Surg 33(1):397–399. 10.1007/s11695-022-06385-2 36469204 10.1007/s11695-022-06385-2 · DOI
  4. [7]Rouhi AD, Ghanem YK, Bader E et al (2023) Online information for incisional hernia repair: what are patients reading? Surgeon 21(4):e195–e200. 10.1016/j.surge.2022.12.002 36588086 10.1016/j.surge.2022.12.002 · DOI
  5. [8]Rouhi AD, Han JJ, Ghanem YK et al (2022) Quality and readability of online patient information on the left ventricular assist device. Artif Organs 47(6):1029–1037. 10.1111/aor.14479 36478254 10.1111/aor.14479 · DOI
  6. [9]Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Quality and readability assessment of online patient information on cytoreductive surgery and hyperthermic intraperitoneal chemotherapy. J Surg Oncol 127(4):699–705. 10.1002/jso.27143 36394434 10.1002/jso.27143 · DOI
  7. [12]Trutner Z, Furlough K, Martinez AB et al (2023) Is health literacy associated with surgical outcomes? A systematic review. J Surg Res 291:720–733. 10.1016/j.jss.2023.06.044 37572516 10.1016/j.jss.2023.06.044 · DOI
  8. [13]Moris D, Paulson EK, Pappas TN (2021) Diagnosis and management of acute appendicitis in adults. JAMA 326(22):2299. 10.1001/jama.2021.20502 34905026 10.1001/jama.2021.20502 · DOI
  9. [14]Charnock D, Shepperd S, Needham G, Gann R (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 53(2):105–111. 10.1136/jech.53.2.105 10396471 10.1136/jech.53.2.105 PMC1756830 · DOI · Paper page
  10. [17]Flesch R (1948) A new readability yardstick. J Appl Psychol 32(3):221–233. 10.1037/h0057532 18867058 10.1037/h0057532 · DOI · Paper page
  11. [19]Massie P, Arshad SA, Auyang ED (2024) Readability of American Society of Metabolic Surgery’s patient information publications. J Surg Res 293:727–732. 10.1016/j.jss.2023.09.018 37862852 10.1016/j.jss.2023.09.018 · DOI
  12. [20]Daraz L, Morrow AS, Ponce OJ et al (2018) Readability of online health information: a meta-narrative systematic review. Am J Med Qual 33(5):487–492. 10.1177/1062860617751639 29345143 10.1177/1062860617751639 · DOI · Paper page
  13. [21]Link E, Baumann E (2020) Nutzung von Gesundheitsinformationen im Internet: personenbezogene und motivationale Einflussfaktoren. Bundesgesundheitsblatt—Gesundheitsforschung—Gesundheitsschutz. 63(6):681–689. 10.1007/s00103-020-03144-5 32367207 10.1007/s00103-020-03144-5 PMC8516774 · DOI
  14. [23]Yeo YH, Samaan JS, Ng WH et al (2023) Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol 29(3):721–732. 10.3350/cmh.2023.0089 36946005 10.3350/cmh.2023.0089 PMC10366809 · DOI
  15. [24]Momenaei B, Wakabayashi T, Shahlaee A et al (2023) Appropriateness and readability of CHATGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmol Retina 7(10):862–868. 10.1016/j.oret.2023.05.022 37277096 10.1016/j.oret.2023.05.022 · DOI
  16. [26]Rao A, Pang M, Kim J et al (2023) Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study. J Med Internet Res 25:e48659. 10.2196/48659 37606976 10.2196/48659 PMC10481210 · DOI · Paper page
  17. [27]Agarwal M, Sharma P, Goswami A (2023) Analysing the applicability of ChatGPT, bard, and bing to generate reasoning-based multiple-choice questions in medical physiology. Cureus. 10.7759/cureus.40977 37519497 10.7759/cureus.40977 PMC10372539 · DOI · Paper page
  18. [28]Giray L (2023) Prompt engineering with ChatGPT: a guide for academic writers. Ann Biomed Eng 51(12):2629–2633. 10.1007/s10439-023-03272-4 37284994 10.1007/s10439-023-03272-4 · DOI · Paper page
  19. [29]Moons P, Van Bulck L (2023) Using ChatGPT and Google Bard to improve the readability of written patient information: a proof of concept. Eur J Cardiovasc Nurs. 10.1093/eurjcn/zvad087 37603843 10.1093/eurjcn/zvad087 · DOI
  20. [30]Meskó B (2023) Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res 25:e50638. 10.2196/50638 37792434 10.2196/50638 PMC10585440 · DOI · Paper page
  21. [31]Emsley R (2023) ChatGPT: these are not hallucinations—they’re fabrications and falsifications. Schizophrenia. 10.1038/s41537-023-00379-4 37598184 10.1038/s41537-023-00379-4 PMC10439949 · DOI · Paper page

Cited by

Sorted by citations

Authors
Ghanem, Y. K., Rouhi, A. D., Al-Houssan, A., Saleh, Z., Moccia, M. C., Joshi, H., Dumon, K. R., Hong, Y., Spitz, F., Joshi, A. R., & Kwiatt, M.
Affiliations
  • Cooper University Hospital (US)
  • University of Pennsylvania (US)
  • University of Connecticut (US)
Corresponding author
Ghanem, Yazid K.
ORCID
Venue
Surgical Endoscopy
Publisher
Springer Science and Business Media LLC
Published
2024-03-05
Volume
38
Issue
5
Pages
2887-2893
ISSN
  • 0930-2794 (ISSN-L)
  • 1432-2218
DOI
10.1007/s00464-024-10739-5
PMC
PMC11078845
OpenAlex
W4392450439
Cited by
97
References
5
Citations by year
2026: 35 · 2025: 46 · 2024: 16
FWCI
9.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Surgical Endoscopy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Rowan University
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Social Sciences › Health › Social Media in Health Education
Keywords
generative AI, AI chatbots, acute appendicitis, online health information, DISCERN, readability, ChatGPT, Google Health, Claude, bards, source verification, health information quality
MeSH terms
Appendicitis, Artificial Intelligence, Humans, Patient Education as Topic, Internet, Comprehension, Consumer Health Information

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review