Proof-of-concept study of a small language model chatbot for breast cancer decision support–a transparent, source-controlled, explainable and data-secure approach
Open accessAbstract
Purpose: Large language models (LLM) show potential for decision support in breast cancer care. Their use in clinical care is currently prohibited by lack of control over sources used for decision-making, explainability of the decision-making process and health data security issues. Recent development of Small Language Models (SLM) is discussed to address these challenges. This preclinical proof-of-concept study tailors an open-source SLM to the German breast cancer guideline (BC-SLM) to evaluate initial clinical accuracy and technical functionality in a preclinical simulation. Methods: A multidisciplinary tumor board (MTB) is used as the gold-standard to assess the initial clinical accuracy in terms of concordance of the BC-SLM with MTB and comparing it to two publicly available LLM, ChatGPT3.5 and 4. The study includes 20 fictional patient profiles and recommendations for 5 treatment modalities, resulting in 100 binary treatment recommendations (recommended or not recommended). Statistical evaluation includes concordance with MTB in % including Cohen’s Kappa statistic (κ). Technical functionality is assessed qualitatively in terms of local hosting, adherence to the guideline and information retrieval. Results: The overall concordance amounts to 86% for BC-SLM (κ = 0.721, p < 0.001), 90% for ChatGPT4 (κ = 0.820, p < 0.001) and 83% for ChatGPT3.5 (κ = 0.661, p < 0.001). Specific concordance for each treatment modality ranges from 65 to 100% for BC-SLM, 85–100% for ChatGPT4, and 55–95% for ChatGPT3.5. The BC-SLM is locally functional, adheres to the standards of the German breast cancer guideline and provides referenced sections for its decision-making. Conclusion: The tailored BC-SLM shows initial clinical accuracy and technical functionality, with concordance to the MTB that is comparable to publicly-available LLMs like ChatGPT4 and 3.5. This serves as a proof-of-concept for adapting a SLM to an oncological disease and its guideline to address prevailing issues with LLM by ensuring decision transparency, explainability, source control, and data security, which represents a necessary step towards clinical validation and safe use of language models in clinical oncology.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hybrid human-AI workflows leverage explanation confidence to triage cases.
“Hybrid human-AI workflows leverage explanation confidence to triage cases: high-confidence predictions with clear explanations may proceed with minimal oversight, while uncertain or poorly-explained cases are routed to human review [91,94].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [94]
The Discussion (Main findings; page unavailable) says the chatbot makes its treatment recommendations traceable by disclosing the guideline sources behind them. It does not describe using explanation confidence to triage cases, allowing high-confidence cases minimal oversight, or routing uncertain cases to human review; the Discussion (Further findings; page unavailable) instead proposes studying hybrid decision-making in future work. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSome explanations cite specific, retrievable evidence, e.g. RAG-grounded source citations or knowledge-graph paths, enabling independent verification.
“Some explanations cite specific, retrievable evidence, e.g. RAG-grounded source citations or knowledge-graph paths, enabling independent verification . This characterizes the RAG-grounded approaches in our review [63, 87,91,92,94,99].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [94]
Retrieval-Augmented Generation can, in principle, ground explanations on retrieved sources, enabling verification against clinical guidelines.
“RetrievalAugmented Generation (RAG) [26] can, in principle, ground explanations on retrieved sources, enabling verification against clinical guidelines [63,94].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“…knowledge graphs and doctoral decision trees of the breast cancer patient pathway into the newly developed SLM design concept to minimize the prevailing challenge of language model hallucination and optimize decision reliability and accuracy (Ji et al. 2022; Benary et al. 2023; Sorin et al. 2024).”
Pembrolizumab for Early Triple-Negative Breast Cancer
Schmid, P., Cortes, J., Pusztai, L., McArthur, H., Kümmel, S., Bergh, J., Denkert, C., Park, Y. H., Hui, R., Harbeck, N., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa1910549 · Cited by 3,270 · cited 3× · see citing works
“Nevertheless, neither does this approach allow for a conclu - sive comparison of treatment modalities or cancer subtypes (i.e., DCIS versus invasive, Her2 positive versus Her2 negative, Luminal A versus Luminal B, TNBC versus HR + or early-stage versus metastatic carcinoma) nor one should expect the cases to produce,…”
InfluentialPrecision medicine, AI, and the future of personalized health care
Johnson, K. B., Wei, W., Weeraratne, D., Frisse, M. E., Misulis, K., Rhee, K., Zhao, J., & Snowdon, J. L. (2021) · Clinical and Translational Science · doi:10.1111/cts.12884 · Cited by 1,854 · see citing works
AI in medicine must be explainable
Kundu, S. (2021) · Nature Medicine · doi:10.1038/s41591-021-01461-z · Cited by 318 · see citing works
Lukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · cited 1× · see citing works
“5 and tumor boards (Lukac et al. 2023; Sorin et al. 2023).”
Flagged first, then by citations
Research works printed but not linked: 27
- Baca SC, Seo JH, Davidsohn MP et al (2023) Liquid biopsy epigenomic profiling for cancer subtyping. Nat Med 29:2737–2741. 10.1038/s41591-023-02605-z 37865722 PMC10695830 · DOI
- Basu P, Mukhopadhyay A, Konishi I (2018) Targeted therapy for gynecologic cancers: toward the era of precision medicine. Int J Gynecol Obstet 143:131–136. 10.1002/ijgo.12620 30306576 · DOI
- Benary M, Wang XD, Schmidt M et al (2023) Leveraging large language models for decision support in personalized oncology. JAMA Netw Open 6:E2343689. 10.1001/jamanetworkopen.2023.43689 37976064 PMC10656647 · DOI · Paper page
- Boca SM, Panagiotou R, Shruti et al (2018) Future of evidence synthesis in precision oncology: between systematic seviews and biocuration. JCO Precis Oncol 2. 10.1200/PO.17.00175 PMC6953752 31930186 · DOI
- Borchert F, Lohr C, Modersohn L et al (2022) GGPONC 2.0-The German clinical guideline corpus for oncology: curation workflow, annotation policy, baseline NER raggers. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3650–3660, Marseille, France. European Language Resources Association
- Choi HS, Song JY, Shin KH et al (2023) Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer. Radiat Oncol J 41:209–216. 10.3857/roj.2023.00633 37793630 PMC10556835 · DOI
- Colomer R, Miranda J, Romero-Laorden N et al (2023) Usefulness and real-world outcomes of next generation sequencing testing in patients with cancer: an observational study on the impact of selection based on clinical judgement. EClinicalMedicine 60. 10.1016/j.eclinm.2023.102029 PMC10248077 37304496 · DOI
- Densen P (2011) Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc 122:48–58 21686208 PMC3116346
- Derks MGM, Bastiaannet E, Kiderlen M et al (2018) Variation in treatment and survival of older patients with non-metastatic breast cancer in five European countries: a population-based cohort study from the EURECCA breast cancer group. Br J Cancer 119:121–129. 10.1038/s41416-018-0090-1 29875471 PMC6035184 · DOI
- Dumontet C, Reichert JM, Senter PD et al (2023) Antibody–drug conjugates come of age in oncology. Nat Rev Drug Discov 22(8):641–661. 10.1038/s41573-023-00709-2 37308581 · DOI
- Gremke N, Rodepeter FR, Teply-Szymanski J et al (2024) NGS-guided precision oncology in breast cancer and gynecological tumors-a retrospective molecular umor board analysis. Cancers (Basel) 16(8):1561. 10.3390/cancers16081561 38672643 PMC11048446 · DOI
- Guo Z, Wang P, Wang Y, Yu S (2023) Improving small Language models on PubMedQA via generative data augmentation. ArXiv. 10.48550/arXiv.2305.0780437873013 · DOI
- Haver HL, Ambinder EB, Bahl M et al (2023) Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology 307. 10.1148/radiol.230424 37014239 · DOI · Paper page
- McIntosh SA, Alam F, Adams L et al (2023) Global funding for cancer research between 2016 and 2020: a content analysis of public and philanthropic investments. Lancet Oncol 24:636–645. 10.1016/S1470-2045(23)00182-1 37269844 · DOI
- Meskó B, Görög M (2020) A short guide for medical professionals in the era of artificial intelligence. NPJ Digit Med 3. 10.1038/s41746-020-00333-z PMC7518439 33043150 · DOI · Paper page
- Porter J, Boyd C, Skandari MR, Laiteerapong N (2023) Revisiting the time needed to provide adult primary care. J Gen Intern Med 38:147–155. 10.1007/s11606-022-07707-x 35776372 PMC9848034 · DOI
- Pritchard KI, Shepherd LE, O’malley FP et al (2006) HER2 and responsiveness of breast cancer to adjuvant chemotherapy. N Engl J Med 354(20):2103–2111. 10.1056/NEJMoa054504 16707747 · DOI
- Rao A, Kim J, Kamineni M et al (2023) Evaluating GPT as an adjunct for radiologic decision making: GPT-4 Versus GPT-3.5 in a breast imaging pilot. J Am Coll Radiol. 10.1016/j.jacr.2023.05.003 37356806 PMC10733745 · DOI
- Schick T, Schütze H (2020) It’s not just size that matters: small language models are also few-shot learners. ArXiv. 10.48550/arXiv.2009.07118 · DOI
- Sorin V, Klang E, Sklair-Levy M et al (2023) Large language model (ChatGPT) as a support tool for breast tumor board. NPJ Breast Cancer 9:44. 10.1038/s41523-023-00557-8 PMC10229606 37253791 · DOI
- Sorin V, Glicksberg BS, Artsi Y et al (2024) Utilizing large language models in breast cancer management: systematic review. J Cancer Res Clin Oncol 150(3):140. 10.1007/s00432-024-05678-6 38504034 PMC10950983 · DOI
- Sosinsky A, Ambrose J, Cross W et al (2024) Insights for precision oncology from the integration of genomic and clinical data of 13,880 tumors from the 100,000 genomes cancer programme. Nat Med 30:279–289. 10.1038/s41591-023-02682-0 38200255 PMC10803271 · DOI
- Subbiah V (2023) The next generation of evidence-based medicine. Nat Med 29:49–58. 10.1038/s41591-022-02160-z 36646803 · DOI
- Taylor C, McGale P, Probert J et al (2023) Breast cancer mortality in 500 000 women with early invasive breast cancer in England, 1993–2015: population based observational cohort study. BMJ. 10.1136/bmj-2022-074684 37311588 PMC10261971 · DOI
- The Lancet Regional Health – Western Pacific (2024) Precision therapeutic landscape for breast cancer: where are we headed? Lancet Reg Health West Pac 42:101024. 10.1016/j.lanwpc.2024.101024 38357396 PMC10865013 · DOI
- Tutt ANJ, Garber JE, Kaufman B et al (2021) Adjuvant olaparib for patients with BRCA1 - or BRCA2 -mutated breast cancer. N Engl J Med 384:2394–2405. 10.1056/nejmoa2105215 34081848 PMC9126186 · DOI · Paper page
- van Walle L, Verhoeven D, Marotti L et al (2023) Trends and variation in treatment of early breast cancer in European certified breast centres: an EUSOMA-based analysis. Eur J Cancer 192. 10.1016/j.ejca.2023.113244 37633095 · DOI
Cited by
Sorted by citations
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
- Authors
- Griewing, S., Lechner, F., Gremke, N., Lukac, S., Janni, W., Wallwiener, M., Wagner, U., Hirsch, M., & Kuhn, S.
- Affiliations
- Philipps University of Marburg (DE)
- Universität Ulm (DE)
- Martin Luther University Halle-Wittenberg (DE)
- Corresponding author
- Griewing, Sebastian
- ORCID
- Venue
- Journal of Cancer Research and Clinical Oncology
- Published
- 2024-10-09
- Volume
- 150
- Issue
- 10
- Pages
- 451-451
- ISSN
- 0084-5353 (ISSN-L)
- 1432-1335
- PMC
- PMC11464535
- OpenAlex
- W4403243803
- Cited by
- 26
- References
- 5
- Citations by year
- 2026: 14 · 2025: 12
- FWCI
- 2.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Journal of Cancer Research and Clinical Oncology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
- Digitalen Hochschulbibliothek Sachsen-Anhalt (Universitäts- und Landesbibliothek Sachsen-Anhalt) (cc-by)
- Funders
- Philipps-Universität Marburg
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- small language models, breast cancer, clinical decision support, multidisciplinary tumor board, ChatGPT, explainability, source control, data security, clinical implementation of AI, open-source AI, clinical validation of AI, AI transparency and accountability
- MeSH terms
- Proof of Concept Study, Breast Neoplasms, Decision Support Techniques, Female, Humans, Practice Guidelines as Topic, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
