Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scientific evaluation of whether LLMs provide a satisfactory level of accuracy for treatment decisions. This observational study compares the concordance of treatment recommendations from the popular LLM ChatGPT 3.5 with those of a multidisciplinary tumor board for breast cancer (MTB). The study design builds on previous findings by combining an extended input model with patient profiles reflecting patho- and immunomorphological diversity of primary breast cancer, including primary metastasis and precancerous tumor stages. Overall concordance between the LLM and MTB is reached for half of the patient profiles, including precancerous lesions. In the assessment of invasive breast cancer profiles, the concordance amounts to 58.8%. Nevertheless, as the LLM makes considerably fraudulent decisions at times, we do not identify the current development status of publicly available LLMs to be adequate as a support tool for tumor boards. Gynecological oncologists should familiarize themselves with the capabilities of LLMs in order to understand and utilize their potential while keeping in mind potential risks and limitations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“Regarding the notion of misalignment and hallucination, research recognizes a major challenge for LLMs, which tend to hallucinate unintended text, limiting their current level of development for use in real-world scenarios [27].”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.03629 · Cited by 4,981
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728 · cited 1×
“…support patients along their care journey; inform and educate patients about their personal cancer diagnosis; facilitate physicians’ access to relevant information by enhancing their up-to-date knowledge; and automate routine medical routine, i.e., automation of discharge summaries [17,22,30,31].”
BackgroundChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708 · cited 1×
“Furthermore, the ongoing diffusion of access and user acceptance of publicly available AI tools, including LLMs like ChatGPT, call for scientific monitoring of how to handle their application in medical care and research [26,29].”
MethodRao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412 · cited 5×
“Furthermore, this level of accuracy meets the average performance of ChatGPT of 71.8% as measured by Rao et al. in their first-of-its-kind study that assessed the AI tool’s potential use along the entire clinical workflow, including diagnostic workup, diagnosis and clinical management [17].”
BackgroundMethodInfluentialEvaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257
Lukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · cited 7×
“[22,23].”
BackgroundMethodResultInfluentialIgnore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · cited 1×
“As the sister-in-law example shows, the stochastic nature of LLMs can be quickly exploited by misaligning simple designed inputs, resulting in fraudulent responses [28].”
Background
Cited by
Large language model integrations in cancer decision-making: a systematic review and meta-analysis
Hao, Y., Qiu, Z., Holmes, J., Löckenhoff, C. E., Liu, W., Ghassemi, M., & Kalantari, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01824-7 · Cited by 67
Griewing, S., Knitza, J., Boekhoff, J., Hillen, C., Lechner, F., Wagner, U., Wallwiener, M., & Kuhn, S. (2024) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-024-07565-4 · Cited by 38
Self-citationArtificial intelligence across the cancer care continuum
Riaz, I. B., Khan, M. A., & Osterman, T. J. (2025) · Cancer · doi:10.1002/cncr.70050 · Cited by 40
Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients
Liao, N., Li, C., Gradishar, W. J., Klimberg, V. S., Roshal, J. A., Yuan, T., Agarwala, S. S., Valero, V. K., Swain, S. M., Margenthaler, J. A., et al. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00001 · Cited by 7
Büyükceran, E. U., Seyfettin, A., Babatürk, A., Özkan, M. B., Çolak, D., Ünal, İ., Kaymaz, E., Ergün, E., Emer, M. Ö., & Mersin, H. H. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02314-9 · Cited by 6
“5 and MTB decisions in breast cancer patients [10], while Sorin et al. observed a 70% agreement rate in a similar context.”
De la Torre-Trillo, J., Yáñez Castillo, Y., Melgarejo Segura, M. T., Carmona Sánchez, E., Zambudio Munuera, A., Mora-Delgado, J., & López Luque, A. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15062130 · Cited by 5
“Similarly, Griewing et al. [28] in breast cancer confirmed that prompt design influences response quality even without advanced techniques.”
Agentic artificial intelligence is the future of cancer detection and diagnosis
Rahman, S., Hosain, M. T., Fahad, N., Morol, M. K., & Hossen, M. J. (2026) · Array · doi:10.1016/j.array.2025.100676 · Cited by 4
AI-assisted tumor board decision-making in pancreatic oncology
Mergen, M., Busch, F., Schwarberg, B., Koldeweihe, P., Jungwirth, D., Sydlik, J., Maurer, H. C., Makowski, M. R., Spitzl, D., & Gassert, F. T. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03444-x · Cited by 3
Metadata
- Authors
- Griewing, S., Gremke, N., Wagner, U., Lingenfelder, M., Kuhn, S., & Boekhoff, J.
- Venue
- Journal of Personalized Medicine
- Publisher
- MDPI AG
- Published
- 2023-10-16
- Volume
- 13
- Issue
- 10
- Pages
- 1502
- ISSN
- 2075-4426 (ISSN-L)
- OpenAlex
- W4387673783
- Cited by
- 77
- References
- 8
- FWCI
- 2.66
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.