Large language models standardize the interpretation of complex oncology guidelines for brain metastases
Open accessPublished version of a preprint
Abstract
The interpretation of nuanced recommendations within complex clinical oncology guidelines, such as those for brain metastases, presents persistent challenges for medical experts, potentially impacting treatment consistency. While Large Language Models offer potential decision support, their comparative efficacy in this domain remains underexplored. This study evaluated the accuracy and convergence of medical experts versus leading Large Language Models in interpreting Strength of Recommendation and Quality of Evidence from the ASTRO and ASCO-SNO-ASTRO brain metastases guidelines. Neurosurgeons, radiation oncologists, and four Large Language Models (ChatGPT-4o, Gemini 2.0, Microsoft Copilot Pro, DeepSeek R1) assessed the Strength of Recommendation and Quality of Evidence for guideline recommendations. Accuracy, near-answer rates, and Cohen’s weighted kappa (κ) were calculated. Large Language Models, notably Gemini and DeepSeek, demonstrate significantly higher accuracy (up to 100% for ASTRO Strength of Recommendation vs. a maximum 58.82% for experts) and near-perfect convergence (κ up to 1.000 vs. κ ≤ 0.504 for experts) in interpreting ASTRO guideline specifics. While all groups found the Quality of Evidence and the more complex ASCO guideline more challenging, Large Language Models generally maintain an advantage in convergence, with Deepseek achieving 61.53% accuracy and κ = 0.428 for ASCO Strength of Recommendation versus a maximum 53.84% accuracy and highly variable convergence for experts. Large Language Models demonstrate significantly higher accuracy than human experts in structured interpretation of guideline classifications, with near-perfect inter- Large Language Model convergence. This supports their role as standardization tools for guideline parsing – freeing experts for patient-specific reasoning where clinical context, comorbidities, and preferences dominate decision-making. Medical experts follow guidelines to treat patients with brain metastases (cancer that has spread to the brain). These guidelines are complex and challenging to interpret consistently. We asked neurosurgeons, radiation oncologists, and four advanced Large Language Models (computer tools) to review and interpret the same guideline recommendations. The Large Language Models are often more accurate and consistent than human experts, especially for specific guideline categories. Our findings suggest that such Large Language Models could assist medical professionals in interpreting complex clinical guideline instructions more accurately and consistently. Yildirim et al. compare the accuracy and consistency of medical experts and Large Language Models in interpreting the details of brain metastases treatment guidelines. Large Language Models demonstrate superior accuracy and better convergence than human readers, highlighting their potential to assist in interpreting standardized guidelines.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Performance of AI guideline navigation remains dependent on input quality and validation.
“Performance remains dependent on input quality and validation.12,13”
From Artificial Intelligence in Oncology: Practical Applications Across Clinical Care, Scholarship, and Translation. (2026) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305 · see citing works
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
Why Don't Physicians Follow Clinical Practice Guidelines?: A Framework for Improvement
Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H., Abboud, P.-A. C., & Rubin, H. R. (1999) · JAMA · doi:10.1001/jama.282.15.1458 · Cited by 6,677 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations
Grimshaw, J., & Russell, I. T. (1993) · The Lancet · doi:10.1016/0140-6736(93)92244-n · Cited by 2,791 · see citing works
Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines
Woolf, S. H., Grol, R. P. T. M., Hutchinson, A., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7182.527 · Cited by 2,549 · see citing works
Clinical guidelines: Developing guidelines
Shekelle, P., Woolf, S. H., Eccles, M., & Grimshaw, J. (1999) · BMJ · doi:10.1136/bmj.318.7183.593 · Cited by 1,220 · see citing works
Progress in evidence-based medicine: a quarter century on
Djulbegovic, B., & Guyatt, G. H. (2017) · The Lancet · doi:10.1016/s0140-6736(16)31592-6 · Cited by 1,022 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Treatment for Brain Metastases: ASCO-SNO-ASTRO Guideline
Vogelbaum, M. A., Brown, P. D., Messersmith, H. J., Brastianos, P. K., Burri, S. H., Cahill, D. P., Dunn, I. F., Gaspar, L. E., Gatson, N. T., Gondi, V., et al. (2021) · Journal of Clinical Oncology · doi:10.1200/jco.21.02314 · Cited by 863 · see citing works
Bias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607 · see citing works
Radiation Therapy for Brain Metastases: An ASTRO Clinical Practice Guideline
Gondi, V., Bauman, G. S., Bradfield, L., Burri, S. H., Cabrera, A. R., Cunningham, D. A., Eaton, B. R., Hattangadi-Gluth, J. A., Kim, M. M., Kotecha, R. R., et al. (2022) · Practical Radiation Oncology · doi:10.1016/j.prro.2022.02.003 · Cited by 417 · see citing works
Artificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review
Shen, J., Zhang, C. J. P., Jiang, B., Chen, J., Song, J., Liu, Z., He, Z., Wong, S. Y., Fang, P.-H., & Ming, W.-K. (2019) · JMIR Medical Informatics · doi:10.2196/10010 · Cited by 373 · see citing works
AI-Driven Clinical Decision Support Systems: An Ongoing Pursuit of Potential
Elhaddad, M., & Hamam, S. (2024) · Cureus · doi:10.7759/cureus.57728 · Cited by 301 · see citing works
Google Gemini as a next generation AI educational tool: a review of emerging educational technology
Imran, M., & Almusharraf, N. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00310-z · Cited by 214 · see citing works
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207 · see citing works
Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study
Wilhelm, T. I., Roos, J., & Kaczmarczyk, R. (2023) · Journal of Medical Internet Research · doi:10.2196/49324 · Cited by 133 · see citing works
Comparative Evaluation of LLMs in Clinical Oncology
Rydzewski, N. R., Dinakaran, D., Zhao, S. G., Ruppin, E., Türkbey, B., Citrin, D. E., & Patel, K. R. (2024) · NEJM AI · doi:10.1056/aioa2300151 · Cited by 108 · see citing works
A Conceptual Framework for Applying Ethical Principles of AI to Medical Practice
Jha, D., Durak, G., Sharma, V., Keleş, E., Çiçek, V., Zhang, Z., Srivastava, A., Rauniyar, A., Hagos, D. H., Tomar, N. K., et al. (2025) · Bioengineering · doi:10.3390/bioengineering12020180 · Cited by 91 · see citing works
Self-citationCook, D. A., Pencille, L. J., Dupras, D. M., Linderbaum, J. A., Pankratz, V. S., & Wilkinson, J. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0191943 · Cited by 81 · see citing works
Brain metastases in metastatic cancer: a review of recent advances in systemic therapies
Steindl, A., & Berghoff, A. S. (2020) · Expert Review of Anticancer Therapy · doi:10.1080/14737140.2021.1851200 · Cited by 25 · see citing works
Zrubka, Z., Champion, A., Holtorf, A., Bidino, R. D., Earla, J. R., Boltyenkov, A., Tabata‐Kelly, M., Asche, C. V., & Burrell, A. D. (2024) · Value in Health · doi:10.1016/j.jval.2024.01.009 · Cited by 25 · see citing works
Assessing the ability of GPT-4o to visually recognize medications and provide patient education
Bazzari, A. H., & Bazzari, F. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-78577-y · Cited by 16 · see citing works
Medicine’s J.A.R.V.I.S. moment: how DeepSeek-R1 transforms clinical practice
Wang, R.-C., He, J.-X., & Liang, H.-R. (2025) · Journal of Thoracic Disease · doi:10.21037/jtd-2025b-05 · Cited by 11 · see citing works
DeepSeek reshaping healthcare in China's tertiary hospitalsPreprint
Chen, J., & Zhang, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.16732 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 1
- [2]Steinberg, E. et al. Clinical Practice Guidelines We Can Trust (National Academies Press, 2011). 24983061
Cited by
Sorted by citations
Hundal, J., Veettil, A. A. V., Chung, C., & Hosseini, M. (2026) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk-26-517280 · Cited by 1 · see citing works
- Authors
- Akkus Yildirim, B., Tutun, B., Durak, G., Yildirim, E. B., Uysal, E., Erturk, S. M., & Bagci, U.
- Affiliations
- Özyeğin University (TR)
- Sağlık Bilimleri Üniversitesi (TR)
- Venue
- Communications Medicine
- Published
- 2025-12-16
- Volume
- 6
- Issue
- 1
- Pages
- 56-56
- ISSN
- 2730-664X (ISSN-L)
- PMC
- PMC12830744
- OpenAlex
- W4417347643
- Cited by
- 2
- References
- 26
- Citations by year
- 2026: 2
- FWCI
- 0.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Communications Medicine (publishedVersion, cc-by-nc-nd)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eResearch@Ozyegin (submittedVersion, cc-by-nc-nd)
- Funders
- Foundation for the National Institutes of Health (R01-HL171376, U01-CA268808)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- large language models, brain metastases, strength of recommendation, quality of evidence, guideline interpretation, ASTRO guidelines, GPT-4o, Gemini, DeepSeek, clinical decision support, interobserver agreement, neurosurgeons, radiation oncologists
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
