Accuracy and consistency of publicly available Large Language Models as clinical decision support tools for the management of colon cancer
Open accessAbstract
Background: Large Language Models (LLM; e.g., ChatGPT) may be used to assist clinicians and form the basis of future clinical decision support (CDS) for colon cancer. The objectives of this study were to (1) evaluate the response accuracy of two LLM‐powered interfaces in identifying guideline‐based care in simulated clinical scenarios and (2) define response variation between and within LLMs. Methods: Clinical scenarios with “next steps in management” queries were developed based on National Comprehensive Cancer Network guidelines. Prompts were entered into OpenAI ChatGPT and Microsoft Copilot in independent sessions, yielding four responses per scenario. Responses were compared to clinician‐developed responses and assessed for accuracy, consistency, and verbosity. Results: Across 108 responses to 27 prompts, both platforms yielded completely correct responses to 36% of scenarios ( n = 39). For ChatGPT, 39% ( n = 21) were missing information and 24% ( n = 14) contained inaccurate/misleading information. Copilot performed similarly, with 37% ( n = 20) having missing information and 28% ( n = 15) containing inaccurate/misleading information ( p = 0.96). Clinician responses were significantly shorter (34 ± 15.5 words) than both ChatGPT (251 ± 86 words) and Copilot (271 ± 67 words; both p < 0.01). Conclusions: Publicly available LLM applications often provide verbose responses with vague or inaccurate information regarding colon cancer management. Significant optimization is required before use in formal CDS.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Performance of AI guideline navigation remains dependent on input quality and validation.
“Performance remains dependent on input quality and validation.12,13”
From Artificial Intelligence in Oncology: Practical Applications Across Clinical Care, Scholarship, and Translation. (2026) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
Accuracy and Reliability of Chatbot Responses to Physician Questions
Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., Berkowitz, S. T., Finn, A. P., Jahangir, E., Scoville, E. A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36483 · Cited by 484 · see citing works
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · cited 1× · see citing works
“29,30 34 The Office of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services (HHS) identified and addressed that transparency of how a model was designed, developed, trained, and evaluated to establish and improve provider trust with use of these…”
Sarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
AI-Driven Clinical Decision Support Systems: An Ongoing Pursuit of Potential
Elhaddad, M., & Hamam, S. (2024) · Cureus · doi:10.7759/cureus.57728 · Cited by 301 · see citing works
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · see citing works
Harnessing the Power of Clinical Decision Support Systems: Challenges and Opportunities
Chen, Z., Liang, N., Zhang, H., Li, H., Yang, Y., Zong, X., Chen, Y., Wang, Y., & Shi, N. (2023) · Open Heart · doi:10.1136/openhrt-2023-002432 · Cited by 244 · see citing works
Boland, G. M., Chang, G. J., Haynes, A. B., Chiang, Y., Chagpar, R., Xing, Y., Hu, C., Feig, B. W., You, Y. N., & Cormier, J. N. (2012) · Cancer · doi:10.1002/cncr.27935 · Cited by 235 · see citing works
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204 · see citing works
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204 · see citing works
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · see citing works
Choudhury, A., & Chaudhry, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/56764 · Cited by 159 · see citing works
Atkins, D., Fink, K., & Slutsky, J. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00002 · Cited by 129 · see citing works
Shamszare, H., & Choudhury, A. (2023) · Healthcare · doi:10.3390/healthcare11162308 · Cited by 113 · see citing works
GPT-4 for Information Retrieval and Comparison of Medical Oncology Guidelines
Ferber, D., Wiest, I. C., Wölflein, G., Ebert, M. P., Beutel, G., Eckardt, J.-N., Truhn, D., Springfeld, C., Jäger, D., & Kather, J. N. (2024) · NEJM AI · doi:10.1056/aics2300235 · Cited by 92 · see citing works
Sallam, M., Barakat, M. M., & Sallam, M. (2023) · Cureus · doi:10.7759/cureus.49373 · Cited by 80 · see citing works
Lee, W. S., Ahn, S., Chung, J., Kim, K. O., Kwon, K. A., Kim, Y., Sym, S.-J., Shin, D., Park, I., Lee, U., et al. (2018) · JCO Clinical Cancer Informatics · doi:10.1200/cci.17.00109 · Cited by 66 · see citing works
Nogueira, L., May, F. P., Yabroff, K. R., & Siegel, R. L. (2023) · Journal of Clinical Oncology · doi:10.1200/jco.23.00539 · Cited by 50 · see citing works
A Survey of Clinicians' Views of the Utility of Large Language Models
Spotnitz, M., Idnay, B., Gordon, E. R., Shyu, R., Zhang, G., Liu, C., Cimino, J. J., & Weng, C. (2024) · Applied Clinical Informatics · doi:10.1055/a-2281-7092 · Cited by 47 · see citing works
ChatGPT, Large Language Models, and Generative AI as Future Augments of Surgical Cancer Care
Kothari, A. N. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13442-2 · Cited by 46 · see citing works
Sblendorio, E., Dentamaro, V., Cascio, A. L., Germini, F., Piredda, M., & Cicolini, G. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105501 · Cited by 45 · see citing works
Zhao, H., Zhang, N., Ho, V., Ding, M., He, W., Niu, J., Yang, M., Du, X. L., Zorzi, D., Chávez‐MacGregor, M., et al. (2017) · Cancer · doi:10.1002/cncr.31094 · Cited by 44 · see citing works
Choo, J. M., Ryu, H. S., Kim, J., Cheong, J. Y., Baek, S. J., Kwak, J., & Kim, J. (2023) · ANZ Journal of Surgery · doi:10.1111/ans.18749 · Cited by 41 · see citing works
Marchi, F., Bellini, E., Iandelli, A., Sampieri, C., & Peretti, G. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08525-z · Cited by 41 · see citing works
Innovating Personalized Nephrology Care: Exploring the Potential Utilization of ChatGPT
Miao, J., Thongprayoon, C., Suppadungsuk, S., Valencia, O. A. G., Qureshi, F., & Cheungpasitporn, W. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13121681 · Cited by 37 · see citing works
Quality of ChatGPT Responses to Questions Related to Pancreatic Cancer and its Surgical Care
Moazzam, Z., Cloyd, J. M., Lima, H. A., & Pawlik, T. M. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13777-w · Cited by 33 · see citing works
McLean, A. L., Wu, Y., McLean, A. C. L., & Hristidis, V. (2024) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-024-05673-x · Cited by 33 · see citing works
Valentini, M., Szkandera, J., Smolle, M. A., Scheipl, S., Leithner, A., & Andreou, D. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1303319 · Cited by 29 · see citing works
Kupfer, S. S., Lubner, S. J., Coronel, E. S., Pickhardt, P. J., Tipping, M. D., Graffy, P. M., Keenan, E., Ross, E. A., Li, T., & Weinberg, D. S. (2018) · Cancer Medicine · doi:10.1002/cam4.1678 · Cited by 26 · see citing works
Strategies for Improving Access to Cancer Services in Rural Communities: A Pre-implementation Study
Crabtree-Ide, C. R., Sevdalis, N., Bellohusen, P., Constine, L. S., Fleming, F. J., Holub, D., Rizvi, I. A., Rodriguez, J., Shayne, M., Termer, N. K., et al. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.818519 · Cited by 26 · see citing works
Kassab, J. T., Dahdah, J. E., Helou, M. C. E., Layoun, H., Sarraju, A., Laffin, L. J., Cho, L., Kapadia, S. R., Collier, P. H., & Harb, S. C. (2023) · Hypertension · doi:10.1161/hypertensionaha.123.21183 · Cited by 26 · see citing works
Applicability of Online Chat-Based Artificial Intelligence Models to Colorectal Cancer Screening
Atarere, J. O., Naqvi, H., Haas, C. J., Adewunmi, C. Y., Bandaru, S. K., Allamneni, R., Ugonabo, O. S., Egbo, O. J., Umoren, M., & Kanth, P. (2024) · Digestive Diseases and Sciences · doi:10.1007/s10620-024-08274-3 · Cited by 26 · see citing works
Evaluation of clinical decision support systems in oncology: An updated systematic review
Nafees, A., Khan, M., Chow, R., Fazelzad, R., Hope, A. J., Liu, G., Létourneau, D., & Raman, S. (2023) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2023.104143 · Cited by 24 · see citing works
Xue, E., Bracken-Clarke, D. M., Iannantuono, G. M., Choo‐Wosoba, H., Gulley, J. L., & Floudas, C. S. (2024) · Journal of Medical Internet Research · doi:10.2196/54758 · Cited by 23 · see citing works
From ChatGPT to Treatment: the Future of AI and Large Language Models in Surgical Oncology
Ramamurthi, A., Are, C., & Kothari, A. N. (2023) · Indian Journal of Surgical Oncology · doi:10.1007/s13193-023-01836-3 · Cited by 19 · see citing works
Chow, Z., Gan, T., Chen, Q., Huang, B., Schoenberg, N. E., Dignan, M., Evers, B. M., & Bhakta, A. S. (2020) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2019.12.041 · Cited by 15 · see citing works
Advancing Medical Practice with Artificial Intelligence: ChatGPT in Healthcare.
Tessler, I., Wolfovitz, A., Livneh, N., Gecel, N. A., Sorin, V., Barash, Y., Konen, E., & Klang, E. (2024) · PubMed · Cited by 15 · see citing works
Nonadherence to Multimodality Cancer Treatment Guidelines in the United States
Tchelebi, L. T., Shen, B., Wang, M., Potters, L., Herman, J. M., Boffa, D. J., Segel, J. E., Park, H. S.-M., & Zaorsky, N. G. (2022) · Advances in Radiation Oncology · doi:10.1016/j.adro.2022.100938 · Cited by 14 · see citing works
Kassab, J. T., Nasr, L. F., Gebrael, G., Helou, M. C. E., Saba, L., Haroun, E., Dahdah, J. E., & Nasr, F. L. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1176617 · Cited by 14 · see citing works
Dawood, Z. S., Hamad, A. B., Moazzam, Z., Alaimo, L., Lima, H. A., Shaikh, C. F., Munir, M. M., Endo, Y., & Pawlik, T. M. (2023) · Surgical Oncology · doi:10.1016/j.suronc.2023.101910 · Cited by 8 · see citing works
Yan, A. P., Parsons, C. R., Caplan, G., Kelly, D. P., Duzan, J., Drake, E., & Kumar, R. (2024) · Pediatric Blood & Cancer · doi:10.1002/pbc.30843 · Cited by 5 · see citing works
Pancreatic Adenocarcinoma: Real World Evidence of Care Delivery in AccessHope Data
Barzi, A., Kim, A. J., Liang, C., West, H., Wong, D. T., Wright, C., Nathwani, N., Vasko, C., Chung, V. M., Rubinson, D. A., et al. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13091377 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [16]Pereyra L, Schlottmann F, Steinberg L, Lasa J. Colorectal Cancer Prevention: Is Chat Generative Pretrained Transformer (Chat GPT) ready to Assist Physicians in Determining Appropriate Screening and Surveillance Recommendations? Journal of Clinical Gastroenterology. 2024. 38319619
Cited by
Sorted by citations
Salman, I. M., Ameer, O. Z., Khanfar, M. A., & Hsieh, Y.-H. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1495378 · Cited by 36 · see citing works
The Use of Artificial Intelligence for Cancer Therapeutic Decision-Making
Elemento, O., Khozin, S., & Sternberg, C. N. (2025) · NEJM AI · doi:10.1056/aira2401164 · Cited by 28 · see citing works
Assessing the accuracy of the GPT-4 model in multidisciplinary tumor board decision prediction
Erdat, E. C., Yalçıner, M., Örüncü, M. B., Ürün, Y., & Şenler, F. Ç. (2025) · Clinical & Translational Oncology · doi:10.1007/s12094-025-03905-1 · Cited by 24 · see citing works
The Role of Artificial Intelligence (ChatGPT-4o) in Supporting Tumor Board Decisions
Karabuğa, B., Karaçin, C., Büyükkör, M., Bayram, D., Aydemir, E., Kaya, O. B., Yılmaz, M. E., Çamöz, E. S., & Ergün, Y. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14103535 · Cited by 22 · see citing works
Iyer, R., Christie, A. P., Madhavapeddy, A., Reynolds, S., Sutherland, W. J., & Jaffer, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0323563 · Cited by 16 · see citing works
Usability of large language models for building construction safety risk assessment
Oral, M., Alboga, Ö., Aydınlı, S., & Erdiş, E. (2025) · Engineering Construction & Architectural Management · doi:10.1108/ecam-08-2024-1143 · Cited by 13 · see citing works
Hack, S., Karni, R. J., Maniaci, A., Fundakowski, C. E., Castellani, L., Incandela, F., Accorona, R., Mayo-Yanez, M., Violati, M., Giannini, L., et al. (2026) · Oral Oncology · doi:10.1016/j.oraloncology.2026.107877 · Cited by 10 · see citing works
Generative artificial intelligence in oncology
Ganjavi, C., Melamed, S., Biedermann, B. M., Eppler, M., Rodler, S., Layne, E., Cei, F., Gill, I. S., & Cacciamani, G. E. (2025) · Current Opinion in Urology · doi:10.1097/mou.0000000000001272 · Cited by 10 · see citing works
Large language models for clinical artificial intelligence in healthcare a systematic review
Ghnemat, R., & Saleh, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00784-x · Cited by 12 · see citing works
The ACPGBI AI taskforce report: A mixed‐methods roadmap for AI in colorectal surgery
Kinross, J. K., Lam, K., Yiu, A. J., Adams, K., Altaf, K., Burns, E. M., Duffourc, M., Eardley, N. J., Evans, C., Giannarou, S., et al. (2025) · Colorectal Disease · doi:10.1111/codi.70232 · Cited by 5 · see citing works
Large Language Models in Colorectal Cancer Care and Clinical Decision Support: Systematic Review
Tian, J., Lou, Q., Wang, X., xuhangying, Mei, H., & Yu, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/89862 · Cited by 4 · see citing works
Li, Z., Liu, H., Tan, W., Tang, D., Duan, S., Zhou, B., Tang, L., Hu, X., Huang, L., Zhao, P., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02645-y · Cited by 5 · see citing works
Yazarkan, Y., Sönmez, G., & Simsek, C. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106413 · Cited by 2 · see citing works
Simsek, C., Vaněk, P., Aydinli, H., Křivinka, J., Lehner, M., Schiavone, S., Hassan, C., & Heinrich, H. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15072692 · Cited by 2 · see citing works
Strugała, A., Batkowski, B., Skrzypoń, K., Mordarski, M., Paniec, P., Chaba-Karnaś, W., Kozioł, N., Czeleń, M., Fibiger, G., Możdżeń, K., et al. (2026) · Clinical Anatomy · doi:10.1002/ca.70189 · Cited by 2 · see citing works
NLP for computational insights into nutritional impacts on colorectal cancer care
Gong, S., Jin, X., Guo, Y., & Yu, J. (2025) · SLAS TECHNOLOGY · doi:10.1016/j.slast.2025.100295 · Cited by 2 · see citing works
Hundal, J., Veettil, A. A. V., Chung, C., & Hosseini, M. (2026) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk-26-517280 · Cited by 1 · see citing works
Can Large Language Models facilitate evidence-based decision support for conservation?Preprint
Iyer, R., Christie, A., Madhavapeddy, A., Reynolds, S., Sutherland, W. J., & Jaffer, S. (2024) · Research Square · doi:10.21203/rs.3.rs-5409185/v1
Iyer, R., Christie, A., Madhavapeddy, A., Reynolds, S., Sutherland, W. J., & Jaffer, S. (2025) · Research Square · doi:10.21203/rs.3.rs-5409185/v2
Aşan, H. (2025) · Uluslararası Sosyal Siyasal ve Mali Araştırmalar Dergisi · doi:10.70101/ussmad.1758962
Large Language Models (LLMs) in Medicine and the Human Role: Between Complement and Transformation
Malek, M. A., & García‐Valiña, L. (2026) · Springer eBooks · doi:10.1007/978-3-031-89820-4_17
Stifini, D., Penna, A. D., Mihaljevic, A., Missios, P., Bitzer, M., Eickhoff, C., & Capobianco, I. (2026) · Research Square · doi:10.21203/rs.3.rs-8849187/v1
Farhadi, A., Amoli, D. M., Fallahzadeh, A., & Zamnifar, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_9
Kara, S., & Karcıoğlu, A. A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12279-6
Si, Y., Gong, S., Thielscher, M., Bateman, H., Li, B., Chen, S., Wang, D., An, R., Meng, Y., Zhang, H., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9988886/v1
Correct but Incomplete: Limitations of AI-Assisted Decision Support in Rectal Cancer
Meyer, R. J., Bresler, T. E., Makaryan, T., Gheissari, R., Mauch, A., & Fujita, M. (2026) · The American Surgeon · doi:10.1177/00031348261471501
Yanagita, T., Saito, T., Tsuru, S., Umeki, R., Sawamura, N., Kurata, S., & Teranishi, F. (2026) · doi:10.2196/preprints.110741
AlSamhori, J. F., AlSamhori, A. R. F., Amourah, R. M., Aldweik, H., AlSamhori, A. F., Holy, L. M., Zakraoui, R., Alomari, S. H., & Shahait, A. D. (2026) · Cureus · doi:10.7759/cureus.116502
- Authors
- Kaiser, K. N., Hughes, A. J., Yang, A. D., Turk, A. A., Mohanty, S., Gonzalez, A. A., Patzer, R. E., Bilimoria, K. Y., & Ellis, R. J.
- Affiliations
- Indiana University School of Medicine (US)
- Indiana University Indianapolis (US)
- Regenstrief Institute (US)
- Corresponding author
- Ellis, Ryan J.
- ORCID
- Venue
- Journal of Surgical Oncology
- Publisher
- Wiley
- Published
- 2024-08-19
- Volume
- 130
- Issue
- 5
- Pages
- 1104-1110
- ISSN
- 0022-4790 (ISSN-L)
- 1096-9098
- PMC
- PMC12049739
- OpenAlex
- W4401709391
- Cited by
- 29
- References
- 40
- Citations by year
- 2026: 17 · 2025: 11 · 2024: 1
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Licence
- Locations
- Journal of Surgical Oncology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- IUScholarWorks (Indiana University) (submittedVersion)
- Funders
- National Institutes of Health
- National Institute on Aging (K23AG071945)
- National Cancer Institute (T32CA282070)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, clinical decision support, ChatGPT, Microsoft Copilot, response accuracy, response consistency, NCCN guidelines, simulated clinical scenarios
- MeSH terms
- Colonic Neoplasms, Humans, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
