Cited by 282Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareBariatric Surgery and Outcomes

Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery

Open access
Samaan, Jamil S., Yeo, Yee Hui, Rajeev, Nithya, Hawley, Lauren, Abel, Stuart, Ng, Wee Han, Srinivasan, Nitin, Park, Justin +4 more·2023·Obesity Surgery

Abstract

PURPOSE: ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery. MATERIALS AND METHODS: Questions were gathered from nationally regarded professional societies and health institutions as well as Facebook support groups. Board-certified bariatric surgeons graded the accuracy and reproducibility of responses. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Reproducibility was determined by asking the model each question twice and examining difference in grading category between the two responses. RESULTS: In total, 151 questions related to bariatric surgery were included. The model provided "comprehensive" responses to 131/151 (86.8%) of questions. When examined by category, the model provided "comprehensive" responses to 93.8% of questions related to "efficacy, eligibility and procedure options"; 93.3% related to "preoperative preparation"; 85.3% related to "recovery, risks, and complications"; 88.2% related to "lifestyle changes"; and 66.7% related to "other". The model provided reproducible answers to 137 (90.7%) of questions. CONCLUSION: The large language model ChatGPT often provided accurate and reproducible responses to common questions related to bariatric surgery. ChatGPT may serve as a helpful adjunct information resource for patients regarding bariatric surgery in addition to standard of care provided by licensed healthcare professionals. We encourage future studies to examine how to leverage this disruptive technology to improve patient outcomes and quality of life.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11695-023-06603-5

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The accuracy and completeness scoring system was based on recommendations from LeapFrog, The Joint Commission, the American College of Surgeons, and relevant literature.

    “To evaluate accuracy and completeness, we developed a scoring system based on recommendations from LeapFrog (VTech Group), The Joint Commission, the American College of Surgeons, and relevant available literature (Multimedia Appendix 2) [21-24].”

    From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [23]

    The Methods > Question Grading section describes a scale that distinguishes accurate, comprehensive answers from correct but incomplete ones. It does not say the scale was based on recommendations from LeapFrog, The Joint Commission, or the American College of Surgeons, or on the cited literature. No page number is available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • The findings of mixed performance align with similar reports of mixed performance across different tasks and clinical text datasets within cancer type-specific domains.

    “These findings align with similar reports of mixed performance across different tasks and clinical text datasets within cancer type-specific domains [45-47]”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., Liran, O., & Samakar, K.
Affiliations
  • Cedars-Sinai Medical Center (US)
  • University of Southern California (US)
  • University of Bristol (GB)
Corresponding author
Samaan, Jamil S.
Venue
Obesity Surgery
Published
2023-04-27
Volume
33
Issue
6
Pages
1790-1796
ISSN
  • 0960-8923 (ISSN-L)
  • 1708-0428
OpenAlex
W4367175507
Cited by
282
References
25
Citations by year
2026: 39 · 2025: 87 · 2024: 115 · 2023: 41
FWCI
11.23
Type
journal article
Language
en
Open access
Locations
  • Obesity Surgery (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Cedars-Sinai Medical Center
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Surgery › Bariatric Surgery and Outcomes
Keywords
ChatGPT, large language models
MeSH terms
Humans, Language, Obesity, Morbid, Quality of Life, Reproducibility of Results, Bariatric Surgery

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review