Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Advances in natural language processing (NLP) and machine learning could assist human users in clinical data extraction from unstructured electronic medical records (EMRs). This study investigates the accuracy and consistency of several Large Language Models (LLMs) - including Dolly, Vicuna, Llama, and GPT-4 - in extracting critical clinical information pertinent to bladder cancer survival prediction. Using EMRs from 163 bladder cancer patients, we assessed the impact on LLM performance by factors such as differences in the trained models, model evolution, input text length, and sequencing of case inputs. GPT-4 demonstrated superior performance with Fleiss' Kappa values exceeding 0.97, accuracy consistently above 93%, and survival prediction metrics closely aligned with ground truth (AUC ± 0.02). Among offline models, Llama-2.0-13b and Llama-3.3-70b exhibited the highest reliability in both information extraction and survival prediction. This study underscores the potential of LLMs to automate clinical data extraction for predictive modeling while highlighting the challenges related to LLM variability and reliability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, it complicates deterministic management of the tool execution process [9].”
From A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“shifting responsibility for data extraction, normalization, and validation directly onto the language model reduces control over the consistency and completeness of the collected information.”
From A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, information extraction performed implicitly by LLMs may exhibit instability across dialogue turns, leading to inconsistent or incomplete tool input data [9,18].”
From A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue (2026) · cites this paper as [9]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
Economic aspects of bladder cancer: what are the benefits and costs?
Sievert, K. D., Amend, B., Nagele, U., Schilling, D., Bedke, J., Horstmann, M., Hennenlotter, J., Kruck, S., & Stenzl, A. (2009) · World Journal of Urology · doi:10.1007/s00345-009-0395-z · Cited by 484
Measuring the Reliability of Qualitative Text Analysis Data
krippendorff, K. (2004) · Quality & Quantity · doi:10.1007/s11135-004-8107-7 · Cited by 411
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01029-4 · Cited by 390
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · cited 1×
“For example, Hu et al. 18 explored prompt engineering to enhance clinical named entity recognition (NER) performance, achieving relaxed F1 scores of 0.794 and 0.861 on the MTSamples dataset with GPT-3.”
InfluentialPrognostic and Prediction Tools in Bladder Cancer: A Comprehensive Review of the Literature
Kluth, L. A., Black, P. C., Bochner, B. H., Catto, J., Lerner, S. P., Stenzl, A., Sylvester, R., Vickers, A. J., Xylinas, E., & Shariat, S. F. (2015) · European Urology · doi:10.1016/j.eururo.2015.01.032 · Cited by 265
Nomograms Provide Improved Accuracy for Predicting Survival after Radical Cystectomy
Shariat, S. F., Karakiewicz, P. I., Palapattu, G. S., Amiel, G. E., Lotan, Y., Rogers, C. G., Vazina, A., Bastian, P. J., Gupta, A., Sagalowsky, A. I., et al. (2006) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-06-0372 · Cited by 263
LLM-based agentic systems in medicine and healthcare
Qiu, J., Lam, K., Li, G., Acharya, A., Wong, T. Y., Darzi, A., Yuan, W., & Topol, E. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00944-1 · Cited by 168
Research works printed but not linked: 3
- Kim, Y. et al. Mdagents: an adaptive collaboration of Llms for medical decision-making. Adv. Neural. Inf. Process. Syst. 37, 79410–79452 (2025).
- Abbasi Yadkori, Y., Kuzborskij, I., György, A. & Szepesvari, C. To believe or not to believe your LLM: iterative prompting for estimating epistemic uncertainty. Adv. Neural. Inf. Process. Syst. 37, 58077–58117 (2025).
- Thatoi, P., Choudhary, R., Shiwlani, A., Qureshi, H. A. & Kumar, S. Natural Language processing (NLP) in the extraction of clinical information from electronic health records (EHRs) for cancer prognosis. Int. J. 10, 2676–2694 (2023).
Cited by
A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue
Pałgan, B., Wawryszczuk, M., Chwał, J., Tkacz, E., & Dzik, R. (2026) · Applied Sciences · doi:10.3390/app16094329 · Cited by 1
Metadata
- Authors
- Sun, D., Hadjiiski, L., Bruno, G., Gormley, J., Chan, H.-P., Caoili, E. M., Cohan, R. H., Alva, A., Mihalcea, R., Zhou, C., & Gulani, V.
- Affiliations
- University of Michigan (US)
- Corresponding author
- Sun, Di
- ORCID
- Venue
- Scientific Reports
- Published
- 2025-11-21
- Volume
- 15
- Issue
- 1
- Pages
- 43773-43773
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC12706081
- OpenAlex
- W4416520295
- Cited by
- 9
- References
- 20
- Citations by year
- 2026: 9
- FWCI
- 3.60
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor, 24 d)
- Locations
- Scientific Reports (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (U01-CA232931)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- large language models, clinical information extraction, bladder cancer, survival prediction, electronic health records, clinical text, LLM reliability, GPT-4, llamas, vicuña, Dolly, Fleiss' kappa, model variability
- MeSH terms
- Machine Learning, Large Language Models, Aged, Urinary Bladder Neoplasms, Female, Humans, Male, Middle Aged, Natural Language Processing, Prognosis, Reproducibility of Results, Data Mining, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis. https://app.cochat.ai/papers/10.1038/s41598-025-27593-7
- The paper
- Sun, D., Hadjiiski, L., Bruno, G., Gormley, J., Chan, H.-P., Caoili, E. M., Cohan, R. H., Alva, A., Mihalcea, R., Zhou, C., & Gulani, V. (2025). Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis. Scientific Reports. https://doi.org/10.1038/s41598-025-27593-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.