CoChat
Sign Up
Cited by 1Artificial Intelligence in Healthcare and EducationColorectal Cancer Surgical TreatmentsClinical practice guidelines implementation

Evaluating the Accuracy of ChatGPT‐4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma

Open access
Meyer, Ryan, Bresler, Tamir E., Palmer, Kevin M., Wilson, Tyler, Pandya, Shivam, Fujita, Manabu·2026·Journal of Surgical Oncology
DOI 10.1002/jso.70314PMC PMC13575598
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: The management of rectal adenocarcinoma requires navigation of complex, branching guideline pathways encompassing neoadjuvant sequencing, surgical approach, organ preservation, and surveillance, yet real‐world guideline adherence remains as low as 60–70%. The ability of current‐generation large language models (LLMs) to accurately navigate these decision points has not been fully characterized. Methods: In this cross‐sectional, vignette‐based study, 135 clinical questions were constructed from 45 pages of NCCN Rectal Cancer Guidelines (Version 4.2024). ChatGPT‐4o was queried using standardized prompts with up to 3 clarifying questions permitted per query. Responses were independently evaluated by two physician raters on a 5‐point Likert scale, with potential discrepancies adjudicated by a board‐certified surgical oncologist. Primary outcomes were the proportion of responses rated Correct (score ≥ 3) and Accurate (score ≥ 4). Inter‐rater reliability was assessed using Cohen's kappa, and subgroup analysis was performed across clinical domains using the Kruskal‐Wallis test. Results: Of 135 questions, 127 (94.1%; 95% CI, 88.7–97.0%) were Correct and 121 (89.6%; 95% CI, 83.3–93.7%) were Accurate. One hundred two responses (75.6%) were completely correct without additional prompting. Performance was consistent across clinical domains (Kruskal‐Wallis H = 0.530, p = 0.767). Inter‐rater agreement was perfect (κ = 1.0). Eight responses (5.9%) contained partially or wholly incorrect information, with errors concentrated in multi‐step conditional treatment decision points. Conclusion: ChatGPT‐4o demonstrates high concordance with NCCN rectal cancer guidelines across all evaluated clinical domains with notable improvement over prior ChatGPT iterations evaluated by our group. The concentration of errors in complex conditional treatment algorithms suggests that LLMs excel at discrete factual recall but may struggle with multi‐step reasoning under clinical uncertainty. Prospective validation using real‐world clinical data and comparison with multidisciplinary tumor board recommendations remain necessary prior to clinical integration.

AI snapshot

Written from the paper’s full text.

TL;DR

ChatGPT-4o answered 135 clinical questions derived from NCCN rectal adenocarcinoma guidelines with 94.1% Correct and 89.6% Accurate responses, with perfect inter-rater agreement. Errors were uncommon (5.9%) but concentrated in multi-step conditional treatment decisions, supporting clinician oversight before clinical integration.

Key findings

  • Of 135 questions, 127 (94.1%; 95% CI, 88.7–97.0%) were rated Correct (score ≥ 3) and 121 (89.6%; 95% CI, 83.3–93.7%) Accurate (score ≥ 4); 102 responses (75.6%) were completely correct without additional prompting.
  • Performance was consistent across clinical domains: Correct rates were 94.6% for workup/diagnostics, 93.5% for treatment, and 95.2% for surveillance; Kruskal-Wallis H = 0.530, p = 0.767; mean scores ranged from 4.52 to 4.61.
  • Eight responses (5.9%) were partially or wholly incorrect (score 1 or 2), including 5 treatment-domain errors involving conditional multi-step algorithms; 2 responses were absolutely incorrect. Inter-rater agreement was perfect (κ = 1.0).
  • The authors report improvement over their prior ChatGPT-4.0 evaluation, which had a 65.8% rate across six gastrointestinal malignancies, though they note direct comparison is limited by different question sets and disease complexity.

Methods

  • Cross-sectional, vignette-based study using 135 clinical questions constructed from 45 pages of NCCN Rectal Cancer Guidelines Version 4.2024, with 3 questions per page assigned to workup/diagnostics, treatment, or surveillance domains.
  • ChatGPT-4o was queried with standardized prompts stating “our institution follows NCCN Guidelines,” each question entered as a separate conversation, and up to 3 standardized clarifying prompts permitted for ambiguous or incomplete responses.
  • Two resident physicians independently rated responses on a 5-point Likert scale; discrepancies were to be adjudicated by a board-certified surgical oncologist. Analyses included Wilson confidence intervals, Cohen’s weighted kappa, and Kruskal-Wallis testing using IBM SPSS Version 29.

Limitations

  • The paper states the 5-point Likert scale remains subjective; controlled vignettes do not capture real-world variability, conflicting information, or unstructured electronic health record data; the 5.9% error rate should be interpreted as a lower bound; raters were aware responses were AI-generated; no human comparator such as a tumor board consensus panel was included; findings are specific to ChatGPT-4o; and the NCCN-context prompt may have improved performance.
  • The text also makes evident that the surveillance subgroup was small (n = 21), producing wider confidence intervals, and that questions were derived directly from specific, well-defined NCCN guideline nodes without diagnostic ambiguity, limiting generalizability to atypical or ambiguous clinical scenarios.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1002/jso.70314

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 16 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Two responses were rated absolutely incorrect (score of 1).

    “Two responses were rated absolutely incorrect (score of 1).”
    Quote in the paperNot a substantial claim
  • Findingown data

    19 responses (14.1%) required clarifying prompts to achieve completeness (score of 4).

    “Nineteen responses (14.1%) required clarifying prompts to achieve completeness (score of 4).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Correct response rate for treatment was 93.5% (95% CI, 85.7-97.2%).

    “Correct response rates were 94.6% (95% CI, 82.3-98.5%)for workup/diagnostics, 93.5% (95% CI, 85.7-97.2%)for treatment, and 95.2% (95% CI, 77.3-99.2%)for surveillance.”
    Quote in the paperSubstantial claim
  • Findingown data

    Eight responses (5.9%) received scores of 1 or 2, indicating partially or wholly incorrect information.

    “Eight responses (5.9%) received scores of 1 or 2, indicating partially or wholly incorrect information (Table 2).”
    Quote in the paperSubstantial claim
  • Findingown data

    102 responses (75.6%) received a score of 5, indicating completely correct answers without any additional prompting.

    “One hundred two responses (75.6%) received a score of 5, indicating completely correct answers without any additional prompting.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 35
  • May combine two claims. 13

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the 135 unique clinical questions, 77 (57.0%) were in the treatment clinical domain.

    “Of the 135 unique clinical questions generated from the NCCN guidelines, 37 (27.4%) were in the workup/diagnostics clinical domain, 77 (57.0%) in treatment, and 21 (15.6%) in surveillance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most common prompt involved directing the model towards NCCN specific recommendations.

    “The most common prompt involved directing the model towards NCCN specific recommendations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Accuracy was consistent across clinical domains, with no statistically significant differences detected (Kruskal-Wallis H = 0.530, df = 2, p = 0.767).

    “Accuracy was consistent across clinical domains, with no statistically significant differences detected (Kruskal-Wallis H = 0.530, df = 2, p = 0.767).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the 8 incorrect responses, 1 involved surveillance.

    “Of these, 5 involved treatment-domain questions (62.5%), 2 involved workup/diagnostics, and 1 involved surveillance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the 8 incorrect responses, 2 involved workup/diagnostics.

    “Of these, 5 involved treatment-domain questions (62.5%), 2 involved workup/diagnostics, and 1 involved surveillance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the 135 unique clinical questions, 37 (27.4%) were in the workup/diagnostics clinical domain.

    “Of the 135 unique clinical questions generated from the NCCN guidelines, 37 (27.4%) were in the workup/diagnostics clinical domain, 77 (57.0%) in treatment, and 21 (15.6%) in surveillance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the 135 unique clinical questions, 21 (15.6%) were in the surveillance clinical domain.

    “Of the 135 unique clinical questions generated from the NCCN guidelines, 37 (27.4%) were in the workup/diagnostics clinical domain, 77 (57.0%) in treatment, and 21 (15.6%) in surveillance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Our controlled vignettes do not fully capture real-world variability.

    “Our controlled vignettes do not fully capture real-world variability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The NCCN Clinical Practice Guidelines for Rectal Cancer provide the most widely adopted evidence-based framework for these decisions in the United States.

    “The National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for Rectal Cancer provide the most widely adopted evidence-based framework for these decisions in the United States”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Inter-rater reliability was assessed using Cohen's weighted kappa.

    “Inter-rater reliability was assessed using Cohen's weighted kappa (κ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Direct comparison is limited by differences in question sets and disease complexity.

    “While direct comparison is limited by differences in question sets and disease complexity, this trend is consistent with the iterative improvements of the omni architecture [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was granted Institutional Review Board exemption due to the absence of human subjects.

    “This study was granted Institutional Review Board exemption due to the absence of human subjects.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    This study was conducted under deliberately optimized conditions.

    “It should be emphasized that this study was conducted under deliberately optimized conditions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    ChatGPT-4o was evaluated on distilled decision points, not on clinical narratives with conflicting information.

    “ChatGPT-4o was evaluated on distilled decision points, not on clinical narratives with conflicting information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    The 5-point Likert scale, though previously reported, remains inherently subjective.

    “Our 5-point Likert scale, though previously reported, remains inherently subjective.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    The 10% gap between the accurate and perfect-score rates, combined with the 5.9% overt error rate, makes clear that LLMs require human oversight, particularly for the conditional treatment pathways.

    “The 10% gap between the accurate and perfect-score rates, combined with the 5.9% overt error rate, makes clear that LLMs require human oversight, particularly for the conditional treatment pathways.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Concordance rates may be lower without this prompt.

    “Concordance rates may be lower without this prompt.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    To reduce memory retention bias, each question was entered as a separate conversation.

    “To reduce memory retention bias, each question was entered as a separate conversation [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Grading incongruities were to be adjudicated by the principal investigator, a board-certified surgical oncologist.

    “Grading incongruities were to be adjudicated by the principal investigator, a board-certified surgical oncologist (M.F.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ChatGPT-4o represents a significant architectural advancement over its predecessor, with reported improvements in context retention, chain-of-thought reasoning, and expanded prompt windows.

    “ChatGPT-4o (”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    ChatGPT-4o achieves high concordance with guideline-based recommendations across workup, treatment, and surveillance domains, with an overall Accuracy rate of 89.6% and Correct rate of 94.1%.

    “Our evaluation of ChatGPT-4o against the NCCN Rectal Cancer guidelines demonstrates that the model achieves high concordance with guideline-based recommendations across workup, treatment, and surveillance domains, with an overall Accuracy rate of 89.6% and Correct rate of 94.1%.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Recent studies have demonstrated promising concordance between LLM recommendations and multidisciplinary tumor board decisions across several malignancies.

    “Recent studies have demonstrated promising concordance between LLM recommendations and multidisciplinary tumor board decisions across several malignancies [7][8]”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [8]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    Our questions were derived directly from NCCN guidelines which are specific and well-defined without any conflicting data or diagnostic ambiguity.

    “Our questions were derived directly from NCCN guidelines which are specific and welldefined without any conflicting data or diagnostic ambiguity.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The NCCN Clinical Practice Guidelines for Rectal Cancer were analyzed in their entirety.

    “The NCCN Clinical Practice Guidelines in Oncology for Rectal Cancer were analyzed in their entirety [2].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Questions were developed by four resident physicians under the guidance of a board-certified surgical oncologist and were designed to test multi-step decision points at specific nodes within the NCCN decision trees.

    “Questions were developed by four resident physicians (K.P., R.M., T.B., T.W.) under the guidance of a board-certified surgical oncologist (M.F.) and were designed to test multi-step decision points at specific nodes within the NCCN decision trees [11][12].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Adjudication by the surgical oncologist was not required for any response.

    “adjudication by the surgical oncologist was not required for any response.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    Clarifying prompts did not contain the correct answer or lead the model toward a predetermined response.

    “Clarifying prompts did not contain the correct answer or lead the model toward a predetermined response.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Our study benefits from a controlled design where every question is mapped to a decision node in the NCCN rectal cancer guidelines.

    “Our study benefits from a controlled design where every question is mapped to a decision node in the NCCN rectal cancer guidelines.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Each of the 135 ChatGPT-4o responses was evaluated independently by two resident physicians against the NCCN guidelines using a 5-point Likert scale.

    “Each of the 135 ChatGPT-4o responses was evaluated independently by two resident physicians (R.M., K.P.) against the NCCN guidelines using the following 5-point Likert scale:”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The error pattern observed (5 of 8 incorrect responses concentrated in treatment-domain questions involving conditional multi-step algorithms) suggests that ChatGPT-4o excels at recalling discrete facts and straightforward pathway branches but may struggle with the conditional logic required when multiple clinical variables must be weighed simultaneously.

    “The error pattern observed in our results (5 of 8 incorrect responses concentrated in treatment-domain questions involving conditional multi-step algorithms) suggests that ChatGPT-4o excels at recalling discrete facts and straightforward pathway branches but may struggle with the conditional logic required when multiple clinical variables must be weighed simultaneously.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Raters scored independently using separate spreadsheets without access to each other's assessments.

    “Raters scored independently using separate spreadsheets without access to each other's assessments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Both raters were aware that responses were AI-generated; this non-blinded design was inherent to the study.

    “Both raters were aware that responses were AI-generated; this non-blinded design was inherent to the study, as the goal was accuracy assessment against a known standard (NCCN guidelines).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    This contextual cue likely improved performance by guiding the model towards specific guidelines.

    “This contextual cue likely improved performance by guiding the model towards specific guidelines.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    ChatGPT-4o achieved high accuracy across all evaluated clinical domains, yet the 5.9% error rate concentrated in complex treatment algorithms reinforces that clinician oversight remains essential.

    “ChatGPT-4o achieved high accuracy across all evaluated clinical domains, yet the 5.9% error rate concentrated in complex treatment algorithms reinforces that clinician oversight remains essential.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Recent efforts to formalize safety and hallucination benchmarking underscore the need for validated scoring systems that extend beyond Likert granularity.

    “recent efforts to formalize safety and hallucination benchmarking underscore the need for validated scoring systems that extend beyond Likert granularity [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A potential source of bias is that raters were aware responses were AI-generated.

    “A potential source of bias is that raters were aware responses were AI-generated, and future studies employing blinded designs could address this.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The clarifying prompt process, while standardized and open-ended, provides the model with additional opportunity to refine its output.

    “The clarifying prompt process, while standardized and open-ended, provides the model with additional opportunity to refine its output”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Without a human benchmark, we cannot determine whether the 5.9% error rate represents a meaningful deficit.

    “Without a human benchmark, we cannot determine whether the 5.9% error rate represents a meaningful deficit.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Our findings are specific to ChatGPT-4o and may not generalize to other LLMs, which differ in training data, architecture, and reasoning capabilities.

    “Our findings are specific to ChatGPT-4o and may not generalize to other LLMs, which differ in training data, architecture, and reasoning capabilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A 2025 cross-sectional study of 610 multidisciplinary tumor board cases reported a mean ChatGPT-4 compatibility score of 3.6/4, with discordance confined largely to rare clinical scenarios.

    “A 2025 cross-sectional study of 610 multidisciplinary tumor board cases reported a mean ChatGPT-4 compatibility score of 3.6/4, with discordance confined largely to rare clinical scenarios [7].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

40 of 40

  • The paper relies on this without support.

    The ability of current-generation LLMs to accurately navigate complex rectal cancer guideline decision points has not been fully characterized.

    “The ability of current-generation large language models (LLMs) to accurately navigate these decision points has not been fully characterized.”

  • The paper relies on this without support.

    These evidence-practice gaps are driven in part by systemic factors including limited access to multidisciplinary tumor boards, resource constraints in rural and community settings, insurance barriers, and the challenge of keeping pace with rapidly evolving treatment recommendations.

    “These evidence-practice gaps are driven in part by systemic factors including limited access to multidisciplinary tumor boards, resource constraints in rural and community settings, insurance barriers, and the challenge of keeping pace with rapidly evolving treatment recommendations.”

  • The paper relies on this without support.

    This mirrors real-world clinical use, where a clinician querying a decision-support tool would provide institutional context.

    “However, this mirrors real-world clinical use, where a clinician querying a decision-support tool would provide institutional context.”

  • The paper relies on this without support.

    LLM performance diminishes on tasks requiring multi-step reasoning under uncertainty.

    “This is consistent with the broader observation that LLM performance diminishes on tasks requiring multi-step reasoning under uncertainty.”

  • The paper relies on this without support.

    Clinicians are most likely to seek AI-assisted decision support in complex, atypical scenarios where the model's performance is not well characterized.

    “Clinicians are most likely to seek AI-assisted decision support in complex, atypical scenarios where the model's performance is not well characterized.”
5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Benson A. B., 2024, Rectal Cancer, Version 1.2024. Nccn Clinical Practice Guidelines in Oncology, Journal of the National Comprehensive Cancer Network: JNCCN, 22, 65

Cited by

Sorted by citations

Related

Authors
Meyer, R., Bresler, T. E., Palmer, K. M., Wilson, T., Pandya, S., & Fujita, M.
Affiliations
Los Robles Hospital & Medical Center (US)
Corresponding author
Meyer, Ryan
ORCID
Venue
Journal of Surgical Oncology
Publisher
Wiley
Published
2026-07-05
Volume
134
Issue
4
Pages
525-530
ISSN
  • 0022-4790 (ISSN-L)
  • 1096-9098
DOI
10.1002/jso.70314
PMC
PMC13575598
OpenAlex
W7167432252
Semantic Scholar
8a52d2b5a123db02b45cacb069e279d0d37b3742
Cited by
1
References
11
Citations by year
2026: 1
FWCI
1.99
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Surgical Oncology (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Oncology › Colorectal Cancer Surgical Treatments
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
GPT-4o, large language models, rectal adenocarcinoma, NCCN guidelines, guideline adherence, organ preservation, postoperative follow-up, surgical approach, AI accuracy evaluation, clinical decision support
MeSH terms
Large Language Models, Generative Artificial Intelligence, Adenocarcinoma, Cross-Sectional Studies, Humans, Rectal Neoplasms, Reproducibility of Results, Practice Guidelines as Topic, Guideline Adherence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-30.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review