Evaluating the performance of ChatGPT in responding to questions about endoscopic procedures for patients
Open accessAbstract
Background and Aims: We evaluated the precision, medical accuracy, superfluous content, and consistency of ChatGPT's responses to commonly asked questions about endoscopic procedures and its capability to provide emotional support, comparing its performance with the generative pretrained transformer 4 (GPT-4) model. Methods: A set of 113 questions related to EGD, colonoscopy, EUS, and ERCP was curated from professional societies and institutional web pages. Responses from ChatGPT were generated and subsequently graded by board-certified gastroenterologists and advanced endoscopists. The emotional support efficacy of ChatGPT and GPT-4 was also assessed by a board-certified psychiatrist (L.S.-M.). Results: ChatGPT exhibited moderate precision in answering questions about EGD (57.9% comprehensive), colonoscopy (47.6% comprehensive), EUS (48.1% comprehensive), and ERCP (44.4% comprehensive). Medical accuracy was highest for EGD (52.6% fully accurate) and lowest for EUS (40.7% fully accurate). Concerning superfluous content, responses were predominantly concise for EGD and colonoscopy, with ERCP and EUS showing increased extraneous content. Reproducibility scores varied across domains, ranging from 50.34% (for EUS) to 68.6% (for EGD). GPT-4 outperformed ChatGPT in emotional support, although both models exhibited satisfactory performance. Conclusions: ChatGPT delivers moderately precise and medically accurate answers related to common endoscopic procedures with varying levels of extraneous content. It holds promise as a supplementary information resource for both patients and healthcare professionals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 1,984 · see citing works
How Internet Users Find, Evaluate, and Use Online Health Information: A Cross-Cultural Review
Morahan-Martin, J. M. (2004) · CyberPsychology & Behavior · doi:10.1089/cpb.2004.7.497 · Cited by 761 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · see citing works
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
Chatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322 · see citing works
Appropriate use of GI endoscopy
Early, D. S., Ben‐Menachem, T., Decker, G. A., Evans, J. A., Fanelli, R. D., Fisher, D. A., Fukami, N., Hwang, J. H., Jain, R., Jue, T. L., et al. (2012) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2012.01.011 · Cited by 318 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139 · see citing works
Zeng, Q. T., Kogan, S., Plovnick, R. M., Crowell, J., Lacroix, E., & Greenes, R. A. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2003.12.015 · Cited by 90 · see citing works
The Influence of Health Literacy on Comprehension of a Colonoscopy Preparation Information Leaflet
Smith, S. G., von Wagner, C., McGregor, L., et al. (2012) · Diseases of the Colon & Rectum · doi:10.1097/dcr.0b013e31826359ac · Cited by 72 · see citing works
Tierney, M., Bevan, R., Rees, C. J., et al. (2015) · Frontline Gastroenterology · doi:10.1136/flgastro-2015-100574 · Cited by 41 · see citing works
Patient Knowledge, Risk Perception, and Barriers to Barrett's Esophagus Screening
Kolb, J. M., Chen, M., Tavakkoli, A., Gallegos, J., O’Hara, J., Tarter, W., Hochheimer, C. J., Golubski, B., Kopplin, N. G., Hennessey, L., et al. (2022) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002054 · Cited by 24 · see citing works
The Difficult-to-Sedate Patient in the Endoscopy Suite
DeLegge, M. H. (2008) · Gastrointestinal Endoscopy Clinics of North America · doi:10.1016/j.giec.2008.06.011 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
- Authors
- Ali, H., Patel, P., Obaitan, I., Mohan, B. P., Sohail, A. H., Smith-Martinez, L., Lambert, K., Gangwani, M. K., Easler, J. J., & Adler, D. G.
- Affiliations
- East Carolina University (US)
- Hofstra University (US)
- Indiana University Indianapolis (US)
- University of Toledo Medical Center (US)
- Porter Adventist Hospital (US)
- Corresponding author
- Adler, Douglas G.
- ORCID
- Venue
- iGIE
- Publisher
- Elsevier BV
- Published
- 2023-12
- Volume
- 2
- Issue
- 4
- Pages
- 553-559
- ISSN
- 2949-7086 (ISSN-L)
- PMC
- PMC12850862
- OpenAlex
- W4387501860
- Cited by
- 15
- References
- 17
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 7
- FWCI
- 0.60
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- iGIE (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- IUScholarWorks (Indiana University) (submittedVersion, cc-by-nc-nd)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- ChatGPT, GPT-4, endoscopy, patient education, colonoscopy, endoscopic ultrasound, endoscopic retrograde cholangiopancreatography, medical accuracy, emotional support, large language models, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
