The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery
Open accessAbstract
PURPOSE: ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery. MATERIALS AND METHODS: Questions were gathered from nationally regarded professional societies and health institutions as well as Facebook support groups. Board-certified bariatric surgeons graded the accuracy and reproducibility of responses. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Reproducibility was determined by asking the model each question twice and examining difference in grading category between the two responses. RESULTS: In total, 151 questions related to bariatric surgery were included. The model provided "comprehensive" responses to 131/151 (86.8%) of questions. When examined by category, the model provided "comprehensive" responses to 93.8% of questions related to "efficacy, eligibility and procedure options"; 93.3% related to "preoperative preparation"; 85.3% related to "recovery, risks, and complications"; 88.2% related to "lifestyle changes"; and 66.7% related to "other". The model provided reproducible answers to 137 (90.7%) of questions. CONCLUSION: The large language model ChatGPT often provided accurate and reproducible responses to common questions related to bariatric surgery. ChatGPT may serve as a helpful adjunct information resource for patients regarding bariatric surgery in addition to standard of care provided by licensed healthcare professionals. We encourage future studies to examine how to leverage this disruptive technology to improve patient outcomes and quality of life.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To evaluate accuracy and completeness, we developed a scoring system based on recommendations from LeapFrog (VTech Group), The Joint Commission, the American College of Surgeons, and relevant available literature (Multimedia Appendix 2) [21-24].”
From Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study (2025) · cites this paper as [23]
Experimental — accuracy not yet measuredIn the citing paper
“These findings align with similar reports of mixed performance across different tasks and clinical text datasets within cancer type-specific domains [45-47]”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Centers for Disease Control and Prevention
Branum, A. (2007) · Sage eBooks · doi:10.4135/9781412952644.n74 · Cited by 6,434
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Benefits and Risks of Bariatric Surgery in Adults: A Review
Arterburn, D. E., Telem, D. A., Kushner, R. F., & Courcoulas, A. P. (2020) · JAMA · doi:10.1001/jama.2020.12567 · Cited by 1,251
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 669
Self-citationOpen artificial intelligence platforms in nursing education: Tools for academic progress or abuse?
O’Connor, S., & ChatGPT (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103537 · Cited by 585
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 466
Theiss, L. M., Wood, T. E., McLeod, M. C., Shao, C. C., Marques, I. C. D. S., Bajpai, S., Lopez, E., Duong, A. M., Hollis, R. H., Morris, M. S., et al. (2021) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2021.10.024 · Cited by 87
Daily briefing: will ChatGPT kill the essay assignment?
Graham, F. (2022) · Nature · doi:10.1038/d41586-022-04437-2 · Cited by 79
Assessment of the quality of Internet information on sleeve gastrectomy
Corcelles, R., Daigle, C. R., Romero‐Talamás, H., Brethauer, S. A., & Schauer, P. R. (2014) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2014.08.014 · Cited by 64
Examination of bariatric surgery Facebook support groups: a content analysis.
Koball, A., Jester, D. J., Domoff, S., Kallies, K. J., Grothe, K. B., & Kothari, S. N. (2017) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2017.04.025 · Cited by 64
Patient-reported health literacy scores are associated with readmissions following surgery
Baker, S. J., Malone, E. B., Graham, L. A., Dasinger, E. A., Wahl, T. S., Titan, A. L., Richman, J., Copeland, L. A., Burns, E. A., Whittle, J., et al. (2020) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2020.06.071 · Cited by 47
Health Literacy and Weight Loss After Bariatric Surgery
Erdoğdu, U. E., Çaycı, H. M., Tardu, A., Demirci, H., Kısakol, G., & Güçlü, M. (2019) · Obesity Surgery · doi:10.1007/s11695-019-04060-7 · Cited by 37
Content and accuracy of nutrition-related posts in bariatric surgery Facebook support groups.
Koball, A., Jester, D. J., Pruitt, M. A., Cripe, R. V., Henschied, J. J., & Domoff, S. (2018) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2018.08.017 · Cited by 35
Readability of online patient-based information on bariatric surgery
Meleo-Erwin, Z. C., Basch, C. H., Fera, J., Ethan, D., & Garcia, P. (2019) · Health Promotion Perspectives · doi:10.15171/hpp.2019.22 · Cited by 35
Facebook groups provide effective social support to patients after bariatric surgery
Athanasiadis, D. I., Roper, A., Hilgendorf, W. A., Voss, A., Zike, T., Embry, M., Banerjee, A., Selzer, D. J., & Stefanidis, D. (2020) · Surgical Endoscopy · doi:10.1007/s00464-020-07884-y · Cited by 35
Batar, N., Kermen, S., Sevdin, S., Yıldız, N., & Güçlü, D. (2020) · Obesity Surgery · doi:10.1007/s11695-020-05015-z · Cited by 35
Patient and the Public's Perceptions of Bariatric Surgery: A Systematic Review
Rajeev, N. D., Samaan, J. S., Premkumar, A., Srinivasan, N., Yu, E., & Samakar, K. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.10.061 · Cited by 33
Self-citationDoes Lower Level of Education and Health Literacy Affect Successful Outcomes in Bariatric Surgery?
Mahoney, S. T., Strassle, P. D., Farrell, T. M., & Duke, M. C. (2019) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1089/lap.2018.0806 · Cited by 28
Bariatric surgery on social media: A cross-sectional study
Pereira, J. P. S., Martinino, A., Manicone, F., Pereira, M. L. S., Iglesias‐Puzas, Á., Pouwels, S., & Mayol, J. M. (2022) · Obesity Research & Clinical Practice · doi:10.1016/j.orcp.2022.02.005 · Cited by 28
Factors Associated With Bariatric Surgery Referral Patterns: A Systematic Review
Premkumar, A., Samaan, J. S., & Samakar, K. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.01.023 · Cited by 23
Self-citationMiller-Matero, L. R., Hecht, L. M., Patel, S. R., Martens, K. M., Hamann, A., & Carlin, A. M. (2020) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2020.09.021 · Cited by 21
Health Literacy, Health Numeracy, and Cognitive Functioning Among Bariatric Surgery Candidates
Hecht, L. M., Cain, S., Clark-Sienkiewicz, S. M., Martens, K. M., Hamann, A., Carlin, A. M., & Miller-Matero, L. R. (2019) · Obesity Surgery · doi:10.1007/s11695-019-04149-z · Cited by 19
The Portrayal of Bariatric Surgery in the UK Print Media
Williamson, J. M., Rink, J., & Hewin, D. H. (2012) · Obesity Surgery · doi:10.1007/s11695-012-0701-5 · Cited by 15
Hecht, L. M., Martens, K. M., Pester, B., Hamann, A., Carlin, A. M., & Miller-Matero, L. R. (2022) · Obesity Surgery · doi:10.1007/s11695-022-05905-4 · Cited by 14
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“, 2023), and many other medical disciplines (Hamidi and Roberts, 2023; Johnson et al., 2023; Samaan et al., 2023), demonstrating its effectiveness in the field of medical queries to a certain extent.”
BackgroundInfluentialA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 329
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 318
Metadata
- Authors
- Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., Liran, O., & Samakar, K.
- Affiliations
- Cedars-Sinai Medical Center (US)
- University of Southern California (US)
- University of Bristol (GB)
- Corresponding author
- Samaan, Jamil S.
- ORCID
- Venue
- Obesity Surgery
- Published
- 2023-04-27
- Volume
- 33
- Issue
- 6
- Pages
- 1790-1796
- ISSN
- 0960-8923 (ISSN-L)
- 1708-0428
- PMC
- PMC10234918
- OpenAlex
- W4367175507
- Cited by
- 282
- References
- 25
- Citations by year
- 2026: 39 · 2025: 87 · 2024: 115 · 2023: 41
- FWCI
- 11.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Obesity Surgery (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Cedars-Sinai Medical Center
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Surgery › Bariatric Surgery and Outcomes
- Keywords
- ChatGPT, large language models
- MeSH terms
- Humans, Language, Obesity, Morbid, Quality of Life, Reproducibility of Results, Bariatric Surgery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
