The performance of artificial intelligence large language model–linked chatbots in surgical decision-making for gastroesophageal reflux disease.
Abstract
Background Large language model (LLM)-linked chatbots may be an efficient source of clinical recommendations for healthcare providers and patients. This study evaluated the performance of LLM-linked chatbots in providing recommendations for the surgical management of gastroesophageal reflux disease (GERD).Methods Nine patient cases were created based on key questions addressed by the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) guidelines for the surgical treatment of GERD. ChatGPT-3.5, ChatGPT-4, Copilot, Google Bard, and Perplexity AI were queried on November 16th, 2023, for recommendations regarding the surgical management of GERD. Accurate chatbot performance was defined as the number of responses aligning with SAGES guideline recommendations. Outcomes were reported with counts and percentages.Results Surgeons were given accurate recommendations for the surgical management of GERD in an adult patient for 5/7 (71.4%) KQs by ChatGPT-4, 3/7 (42.9%) KQs by Copilot, 6/7 (85.7%) KQs by Google Bard, and 3/7 (42.9%) KQs by Perplexity according to the SAGES guidelines. Patients were given accurate recommendations for 3/5 (60.0%) KQs by ChatGPT-4, 2/5 (40.0%) KQs by Copilot, 4/5 (80.0%) KQs by Google Bard, and 1/5 (20.0%) KQs by Perplexity, respectively. In a pediatric patient, surgeons were given accurate recommendations for 2/3 (66.7%) KQs by ChatGPT-4, 3/3 (100.0%) KQs by Copilot, 3/3 (100.0%) KQs by Google Bard, and 2/3 (66.7%) KQs by Perplexity. Patients were given appropriate guidance for 2/2 (100.0%) KQs by ChatGPT-4, 2/2 (100.0%) KQs by Copilot, 1/2 (50.0%) KQs by Google Bard, and 1/2 (50.0%) KQs by Perplexity.Conclusions Gastrointestinal surgeons, gastroenterologists, and patients should recognize both the promise and pitfalls of LLM's when utilized for advice on surgical management of GERD. Additional training of LLM's using evidence-based health information is needed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1× · see citing works
“Many online news reports state that ChatGPT and other LLMs may provide comparable health management to a physician, largely based on studies showing that they can pass licensing examinations [25, 26], or even respond to patient inquiries with greater empathy than clinicians [5].”
Update on the epidemiology of gastro-oesophageal reflux disease: a systematic review
El‐Serag, H. B., Sweet, S., Winchester, C., & Dent, J. (2013) · Gut · doi:10.1136/gutjnl-2012-304269 · Cited by 1,934 · see citing works
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · see citing works
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840 · see citing works
ChatGPT and large language models in academia: opportunities and challenges
Meyer, J. G., Urbanowicz, R. J., Martin, P. C. N., O’Connor, K., Li, R., Peng, P.-C., Bright, T. J., Tatonetti, N., Won, K. J., Gonzalez-Hernandez, G., et al. (2023) · BioData Mining · doi:10.1186/s13040-023-00339-9 · Cited by 569 · see citing works
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541 · see citing works
Beck, F., Richard, J., Nguyen‐Thanh, V., Montagni, I., Parizot, I., & Renahy, É. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2934 · Cited by 363 · see citing works
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · see citing works
Amante, D. J., Hogan, T. P., Pagoto, S., English, T. M., & Lapane, K. L. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4126 · Cited by 350 · see citing works
Lee, K., Hoti, K., Hughes, J. D., & Emmerton, L. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3706 · Cited by 284 · see citing works
How AI Responds to Common Lung Cancer Questions: ChatGPT versus Google Bard
Rahsepar, A. A., Tavakoli, N., Kim, G. H. J., Hassani, C., Abtin, F. G., & Bedayat, A. (2023) · Radiology · doi:10.1148/radiol.230922 · Cited by 283 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
SAGES guidelines for the surgical treatment of gastroesophageal reflux (GERD)
Slater, B. J., Dirks, R. C., McKinley, S. K., Ansari, M., Kohn, G. P., Thosani, N. C., Qumseya, B. J., Billmeier, S. E., Daly, S. C., Crawford, C., et al. (2021) · Surgical Endoscopy · doi:10.1007/s00464-021-08625-5 · Cited by 138 · see citing works
Self-citationEight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137 · cited 1× · see citing works
“While Rahsepar et al. reported that the converse was true when evaluating lung cancer screening recommendations provided by these LLMs [22], computer scientists recognize that reinforcement learning, as described in creating the scenarios, can both strength and weaken model behaviors [23].”
Mahajan, A., Esper, S. A., Oo, T. H., McKibben, J., Garver, M., Artman, J., Klahre, C., Ryan, J. P., Sadhasivam, S., Holder‐Murray, J., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.22285 · Cited by 112 · see citing works
Situating Wikipedia as a health information resource in various contexts: A scoping review
Smith, D. A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0228786 · Cited by 104 · see citing works
Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions
Ayoub, N., Lee, Y.-J., Grimm, D. R., & Balakrishnan, K. (2023) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2023.0704 · Cited by 98 · see citing works
Gastroesophageal reflux disease: A review of surgical decision making
Moore, M. D. (2016) · World Journal of Gastrointestinal Surgery · doi:10.4240/wjgs.v8.i1.77 · Cited by 93 · see citing works
ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results
Hristidis, V., Ruggiano, N., Brown, E. L., Ganta, S. R. R., & Stewart, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48966 · Cited by 93 · see citing works
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83 · see citing works
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S. M., Özkan, E., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103042 · Cited by 55 · see citing works
Online information search behaviour of physicians
Mikalef, P., Kourouthanassis, P. E., & Pateli, A. G. (2017) · Health Information & Libraries Journal · doi:10.1111/hir.12170 · Cited by 54 · see citing works
Reporting standards for the use of large language model-linked chatbots for health advice.
Huo, B., Cacciamani, G. E., Collins, G. S., McKechnie, T., Lee, Y., & Guyatt, G. (2023) · Nature Medicine · doi:10.1038/s41591-023-02656-2 · Cited by 53 · see citing works
Self-citationHenson, J. B., Brown, J. R. G., Lee, J. P., Patel, A. C., & Leiman, D. A. (2023) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002397 · Cited by 41 · see citing works
Emile, S. H., Ghareeb, W. M., Elfeki, H. A., et al. (2022) · Obesity Surgery · doi:10.1007/s11695-022-06112-x · Cited by 40 · see citing works
User preferences for ChatGPT-powered conversational interfaces versus traditional methods
Sakirin, T., & Said, R. B. (2023) · Mesopotamian Journal of Computer Science · doi:10.58496/mjcsc/2023/004 · Cited by 25 · see citing works
Ge, Z., Wang, B., Chang, J., et al. (2023) · Scandinavian Journal of Gastroenterology · doi:10.1080/00365521.2022.2163185 · Cited by 23 · see citing works
Kamiński, M., Łoniewski, I., Misera, A., & Marlicz, W. M. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16234591 · Cited by 17 · see citing works
Sachs, G. F., Ourshalimian, S. A., Jensen, A. R., et al. (2023) · Surgery · doi:10.1016/j.surg.2023.07.008 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Self-citationRossettini, G., Bargeri, S., Cook, C., Guida, S., Palese, A., Rodeghiero, L., Pillastrini, P., Turolla, A., Castellini, G., & Gianola, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1574287 · Cited by 31 · see citing works
- Authors
- Huo, B., Calabrese, E., Sylla, P., Kumar, S., Ignacio, R. C., Oviedo, R., Hassan, I., Slater, B. J., Kaiser, A., Walsh, D. S., & Vosburg, W.
- Affiliations
- McMaster University (CA)
- University of California System (US)
- Icahn School of Medicine at Mount Sinai (US)
- Thomas Jefferson University Hospital (US)
- University of California San Diego (US)
- Sam Houston State University (US)
- University of Iowa (US)
- University of Chicago (US)
- City Of Hope National Medical Center (US)
- University of Kentucky (US)
- ORCID
- Venue
- Surgical Endoscopy
- Published
- 2024-04-17
- Volume
- 38
- Issue
- 5
- Pages
- 2320-2330
- ISSN
- 0930-2794 (ISSN-L)
- 1432-2218
- OpenAlex
- W4394873790
- Cited by
- 28
- References
- 31
- Citations by year
- 2026: 11 · 2025: 13 · 2024: 4
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Surgical Endoscopy (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, chatbots, gastroesophageal reflux disease, ChatGPT-4, Google Bard, Copilot, Perplexity AI, guideline adherence, pediatric GERD, LLM evaluation, AI hallucination
- MeSH terms
- Clinical Decision-Making, Adult, Artificial Intelligence, Gastroesophageal Reflux, Humans, Male, Practice Guidelines as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
