Accuracy and Reproducibility of ChatGPT Responses to Breast Cancer Tumor Board Patients
Abstract
PURPOSE: We assessed the accuracy and reproducibility of Chat Generative Pre-Trained Transformer's (ChatGPT) recommendations in response to breast cancer patients by comparing generated outputs with consensus expert opinions. METHODS: 362 consecutive breast cancer patients sourced from a weekly international breast cancer webinar series were submitted to a tumor board of renowned experts. The same 362 clinical patients were also prompted to ChatGPT-4.0 three separate times to examine reproducibility. RESULTS: Only 46% of ChatGPT-generated content was entirely concordant with the recommendations of breast cancer experts, and only 39% of ChatGPT's responses demonstrated inter-response similarity. ChatGPT's responses demonstrated higher concordance with CEN experts in earlier stages of breast cancer (0, I, II, III) compared to advanced (IV) patients ( P = .019). There were less accurate responses from ChatGPT when responding to patients involving molecular markers and genetic testing ( P = .025), and in patients involving antibody drug conjugates ( P = .006). ChatGPT's responses were not necessarily incorrect but often omitted specific details about clinical management. When the same prompt was independently sent to CEN into the model on three occasions, each time by difference users, ChatGPT's responses exhibited variable content and formatting in 68% (246 out of 362) of patients and were entirely consistent with one another in only 32% of responses. CONCLUSION: Since this promising clinical decision-making support tool is widely used currently by physicians worldwide, it is important for the user to understand its limitations as currently constructed when responding to multidisciplinary breast cancer patients, and for researchers in the field to continue improving its ability with contemporary, accurate and complete breast cancer information. As currently constructed, ChatGPT is not engineered to generate identical outputs to the same input and was less likely to correctly interpret and recommend treatments for complex breast cancer patients.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Liao et al. reported that only 46% of ChatGPT-4.0 outputs fully aligned with expert recommendations across 362 patients, with lower performance in advanced-stage disease and molecular marker-dependent scenarios.
“In breast cancer, Liao et al. reported that only 46% of ChatGPT-4.0 outputs fully aligned with expert recommendations across 362 patients, with lower performance in advanced-stage disease and molecular marker-dependent scenarios [20].”
Liao et al. reported 32% consistency in breast cancer.
“closely paralleling the finding of 32% consistency reported by Liao et al. in breast cancer [20].”
Genomic aspects of decision-making represent a particularly vulnerable domain for current LLMs.
“suggesting that genomic aspects of decision-making represent a particularly vulnerable domain for current LLMs.”
Our concordance rates are consistent with findings from studies in similarly complex clinical settings.
“Our concordance rates are consistent with findings from studies in similarly complex clinical settings.”
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Lukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Asoglu, H., Kose, S., Kayim, O., Abaci, A., Khatib, G., Vardar, M. A., Kilic Bagir, E., Gumurdulu, D., Kidi, M. M., Bayram, E., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15124451 · Cited by 1 · see citing works
“0 outputs fully aligned with expert recommendations across 362 patients, with lower performance in advanced-stage disease and molecular marker-dependent scenarios [20].”
Influential
- Authors
- Liao, N., Li, C., Gradishar, W. J., Klimberg, V. S., Roshal, J. A., Yuan, T., Agarwala, S. S., Valero, V. K., Swain, S. M., Margenthaler, J. A., Rubio, I. T., Hurvitz, S. A., Geyer, C. E., Lin, N. U., Rugo, H. S., Zhang, G., Liu, N., & Balch, C. M.
- Affiliations
- Guangdong Academy of Medical Sciences (CN)
- Northwestern University (US)
- The University of Texas MD Anderson Cancer Center (US)
- The University of Texas Medical Branch at Galveston (US)
- Guangzhou Concord Cancer Center, Guangzhou, China
- Cancer Expert Now Inc, Morristown, NJ
- Georgetown University Medical Center (US)
- Washington University in St. Louis (US)
- Clinica Universidad de Navarra (ES)
- Fred Hutch Cancer Center (US)
- University of Pittsburgh (US)
- Dana Farber Cancer Institute, Boston, MA
- University of California, San Francisco (US)
- ORCID
- Venue
- JCO Clinical Cancer Informatics
- Published
- 2025-06
- Volume
- 9
- Issue
- 9
- Pages
- e2500001-e2500001
- ISSN
- 2473-4276 (ISSN-L)
- OpenAlex
- W4411017059
- Cited by
- 7
- References
- 3
- Citations by year
- 2026: 4 · 2025: 3
- FWCI
- 1.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JCO Clinical Cancer Informatics (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- ChatGPT, large language models, breast cancer, multidisciplinary tumor board, clinical decision support, AI accuracy, reproducibility, molecular markers, genetic testing, antibody-drug conjugates, expert consensus, stage at diagnosis
- MeSH terms
- Generative Artificial Intelligence, Breast Neoplasms, Female, Humans, Middle Aged, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
