Communicative competence of generative artificial intelligence in responding to patient queries about colorectal cancer surgery
Open accessAbstract
Purpose: To examine the ability of generative artificial intelligence (GAI) to answer patients’ questions regarding colorectal cancer (CRC). Methods: Ten clinically relevant questions about CRC were selected from top-rated hospitals’ websites and patient surveys and presented to three GAI tools (Chatbot Generative Pre-Trained Transformer [GPT-4], Google Bard, and CLOVA X). Their responses were compared with answers from the CRC information book. Response evaluation was performed by two groups, each consisting of five healthcare professionals (HCP) and patients. Each question was scored on a 1–5 Likert scale based on four evaluation criteria (maximum score, 20 points/question). Results: In an analysis including only HCPs, the information book scored 11.8 ± 1.2, GPT-4 scored 13.5 ± 1.1, Google Bard scored 11.5 ± 0.7, and CLOVA X scored 12.2 ± 1.4 ( P = 0.001). The score of GPT-4 was significantly higher than those of the information book ( P = 0.020) and Google Bard ( P = 0.001). In an analysis including only patients, the information book scored 14.1 ± 1.4, GPT-4 scored 15.2 ± 1.8, Google Bard scored 15.5 ± 1.8, and CLOVA X scored 14.4 ± 1.8, without significant differences ( P = 0.234). When both groups of evaluators were included, the information book scored 13.0 ± 0.9, GPT-4 scored 14.4 ± 1.2, Google Bard scored 13.5 ± 1.0, and CLOVA X scored 13.3 ± 1.5 ( P = 0.070). Conclusion: The three GAIs demonstrated similar or better communicative competence than the information book regarding questions related to CRC surgery in Korean. If high-quality medical information provided by GAI is supervised properly by HCPs and published as an information book, it could be helpful for patients to obtain accurate information and make informed decisions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies demonstrated that generative AI tools tailor educational content based on patient demographics, clinical context, and informational needs, including evaluations of ChatGPT and other LLMs responding to cancer-related questions and decision-making scenarios.
“Recent studies further extend this work by demonstrating how generative AI tools tailor educational content based on patient demographics, clinical context, and informational needs, including evaluations of ChatGPT and other large language models (LLMs) responding to cancer-related questions and decision-making scenarios [22-25].”
From Empathic AI for Patient-Centered Cancer Care: A Scoping Review of Patient Navigation, Support, and Clinical Practice (2026) · cites this paper as [22]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 3× · see citing works
“…and the grammatical appropriateness of the answer) [14], level of empathy (quantity and quality of empathetic expressions in the response) [7], humanness of the answer (similarity to linguistic expressions used in actual human responses) [15], and patient satisfaction (understanding of…”
BackgroundInfluentialJohnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
Self-citationTaira, K., Itaya, T., & Hanada, A. (2023) · JMIR Nursing · doi:10.2196/47305 · Cited by 76 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- [1]Taylor K (2009) Paternalism, participation and partnership – the evolution of patient centeredness in the consultation. Patient Educ Couns 74:150–155. 10.1016/j.pec.2008.08.017 18930624 · DOI
- [2]Fernández-Ballesteros R, Sánchez-Izquierdo M, Olmos R, Huici C, Ribera Casado JM, Cruz Jentoft A (2019) Paternalism vs. autonomy: are they alternative types of formal care? Front Psychol 10:1460. 10.3389/fpsyg.2019.01460 31316428 PMC6611139 · DOI
- [3]Entwistle VA, Carter SM, Cribb A, McCaffery K (2010) Supporting patient autonomy: the importance of clinician-patient relationships. J Gen Intern Med 25:741–745. 10.1007/s11606-010-1292-2 20213206 PMC2881979 · DOI
- [4]Kilbride MK, Joffe S (2018) The new age of patient autonomy: implications for the patient-physician relationship. JAMA 320:1973–1974. 10.1001/jama.2018.14382 30326026 PMC6988779 · DOI
- [5]Sauerbrei A, Kerasidou A, Lucivero F, Hallowell N (2023) The impact of artificial intelligence on the person-centred, doctor-patient relationship: some problems and solutions. BMC Med Inform Decis Mak 23:73. 10.1186/s12911-023-02162-y 37081503 PMC10116477 · DOI
- [8]Health insurance review & assessment service search for hospitals’ national quality assessments. Internet. Health Insurance Review & Assessment Service; c2023. https://www.hira.or.kr/ra/eval/getDiagEvlList.do?pgmid=HIRAA030004000100. Accessed 5 Dec 2023.
- [9]Rahsepar AA, Tavakoli N, Kim GHJ, Hassani C, Abtin F, Bedayat A (2023) How AI responds to common lung cancer questions: ChatGPT vs Google Bard. Radiology 307:e230922. 10.1148/radiol.230922 37310252 · DOI
- [10]Tariq R, Malik S, Khanna S (2024) Evolving landscape of large language models: an evaluation of ChatGPT and Bard in answering patient queries on colonoscopy. Gastroenterology 166:220–221. 10.1053/j.gastro.2023.08.033 37634736 · DOI
- [11]Lim ZW, Pushpanathan K, Yew SM, Lai Y, Sun CH, Lam JS, Chen DZ, Goh JH, Tan MC, Sheng B, Cheng CY (2023) Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBiomedicine 95:104770. 10.1016/j.ebiom.2023.104770 37625267 PMC10470220 · DOI
- [12]Coskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y (2024) Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int 44:509–515. 10.1007/s00296-023-05473-5 37747564 · DOI
- [14]Vaira LA, Lechien JR, Abbate V, Allevi F, Audino G, Beltramini GA, Bergonzani M, Bolzoni A, Committeri U, Crimi S, Gabriele G, Lonardi F, Maglitto F, Petrocelli M, Pucci R, Saponaro G, Tel A, Vellone V, Chiesa-Estomba CM et al (2024) Epub Accuracy of ChatGPT-generated information on head and neck and oromaxillofacial surgery: A multicenter collaborative analysis. Otolaryngol Head Neck Surg 170:1492–1503. 10.1002/ohn.489 37595113 · DOI
- [15]Ali SR, Dobbs TD, Hutchings HA, Whitaker IS (2023) Using ChatGPT to write patient clinic letters. Lancet Digit Health 5:e179–e181. 10.1016/S2589-7500(23)00048-1 36894409 · DOI · Paper page
- [17]Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, Almeida D, Altenschmidt J, Altman S, Anadkat S, Avila R (2023) Gpt-4 technical report. arXiv preprint arXiv. 10.48550/arXiv.2303.08774 · DOI · Paper page
- [18]Varghese J, Chapiro J (2024) ChatGPT: the transformative influence of generative AI on science and healthcare. J Hepatol 80:977–980. 10.1016/j.jhep.2023.07.028 37544516 · DOI
- [19]Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Zadnik Sullivan PL, Cielo D, Oyelese AA, Doberstein CE, Telfeian AE, Gokaslan ZL, Asaad WF (2023) Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery 93:1090–1098. 10.1227/neu.0000000000002551 37306460 · DOI
- [21]Thoppilan R, De Freitas D, Hall J, Shazeer N, Kulshreshtha A, Cheng HT, Jin A, Bos T, Baker L, Du Y, Li Y (2022) Lamda: language models for dialog applications. arXiv preprint arXiv. 10.48550/arXiv.2201.08239 · DOI · Paper page
- [23]Kim B, Kim H, Lee SW, Lee G, Kwak D, Jeon DH, Park S, Kim S, Kim S, Seo D, Lee H (2021) What changes can large-scale language models bring? intensive study on hyperclova: billions-scale Korean generative pretrained transformers. arXiv preprint arXiv. 10.48550/arXiv.2109.04650 · DOI
- [24]Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D (2023) How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 9:e45312. 10.2196/45312 36753318 PMC9947764 · DOI · Paper page
- [26]Cai LZ, Shaheen A, Jin A, Fukui R, Yi JS, Yannuzzi N, Alabiad C (2023) Performance of generative large language models on ophthalmology board-style questions. Am J Ophthalmol 254:141–149. 10.1016/j.ajo.2023.05.024 37339728 · DOI · Paper page
- [28]Lahat A, Shachar E, Avidan B, Glicksberg B, Klang E (2023) Evaluating the utility of a large language model in answering common patients’ gastrointestinal health-related questions: are we there yet? Diagnostics (Basel) 13. 10.3390/diagnostics13111950 PMC10252924 37296802 · DOI
- [29]Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A (2023) Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol 29:721–732. 10.3350/cmh.2023.0089 36946005 PMC10366809 · DOI
Cited by
Sorted by citations
White, B. M., Aitken, G. L., Zink, J. A., Kheirkhah Rahimabad, P., Kumsa, F. A., Hashtarkhani, S., Rashid, R., Kheirinejad, S., Girdwood, T., Brett, C. L., et al. (2026) · JMIR Cancer · doi:10.2196/82336 · Cited by 2 · see citing works
“Other researchers have used NLP approaches, such as analyzing emotion intensity or identifying psychosocial challenges, to address end-of-life discussions with patients with cancer and provide them with targeted resources specific to their stage of the cancer journey [22,24,35,36,41,56].”
Influential
- Authors
- Jo, M. H., Kim, M.-J., Oh, H.-K., Choi, M. J., Shin, H.-R., Lee, T.-G., Ahn, H.-M., Kim, D.-W., & Kang, S.-B.
- Affiliations
- Seoul National University Bundang Hospital (KR)
- Seoul National University (KR)
- Corresponding author
- Oh, Heung-Kwon
- ORCID
- Venue
- International Journal of Colorectal Disease
- Published
- 2024-06-20
- Volume
- 39
- Issue
- 1
- Pages
- 94-94
- ISSN
- 0179-1958 (ISSN-L)
- 1432-1262
- PMC
- PMC11189990
- OpenAlex
- W4399848185
- Cited by
- 17
- References
- 4
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 2
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Colorectal Disease (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- generative AI, colorectal cancer surgery, patient education, patient education materials, GPT-4, Google Bard, patient-provider communication, health information quality, informed decision-making
- MeSH terms
- Artificial Intelligence, Colorectal Surgery, Communication, Female, Humans, Male, Middle Aged, Surveys and Questionnaires, Colorectal Neoplasms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
