CoChat

Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis

Open access

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

Advances in natural language processing (NLP) and machine learning could assist human users in clinical data extraction from unstructured electronic medical records (EMRs). This study investigates the accuracy and consistency of several Large Language Models (LLMs) - including Dolly, Vicuna, Llama, and GPT-4 - in extracting critical clinical information pertinent to bladder cancer survival prediction. Using EMRs from 163 bladder cancer patients, we assessed the impact on LLM performance by factors such as differences in the trained models, model evolution, input text length, and sequencing of case inputs. GPT-4 demonstrated superior performance with Fleiss' Kappa values exceeding 0.97, accuracy consistently above 93%, and survival prediction metrics closely aligned with ground truth (AUC ± 0.02). Among offline models, Llama-2.0-13b and Llama-3.3-70b exhibited the highest reliability in both information extraction and survival prediction. This study underscores the potential of LLMs to automate clinical data extraction for predictive modeling while highlighting the challenges related to LLM variability and reliability.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-025-27593-7

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Heatmap representing the frequency of extracted information for the three clinical variables (pT, pN, and LVI) across the 10 runs (N = 163) for the six LLMs: Dolly-v2-12b, Vicuna-v1.3-13b, Llama-2.0-13b, Llama-3.3-70b, GPT-4 Turbo, GPT-4o. The input for all the six models was the snippet reports and in their original sequence. The y-axis displays the categories for the specified clinical variables: pT (N/A, invalid, pT0, pTis, pTa, pT1, pT2, pT3, pT4), pN (N/A, invalid, N0, N1, N2, N3), and LVI (N/A, invalid, Yes, No), in which ‘N/A’ indicates that relevant information in the input report for LLMs was unavailable or absent, and ‘invalid’ signifies that the LLM’s output could not be interpreted as meaningful or valid for the given variable. The x-axis corresponds to individual cases. Color intensity indicates the frequency of each category in the data. The color bar on the right provides a reference for the frequency scale. Orange rectangles highlight the ground truth values for each case, enabling comparison with the LLM extractions. (a) Dolly-v2-12b, (b) Vicuna-v1.3-13b, (c) Llama-2.0-13b, (d) Llama-3.3-70b, (e) GPT-4 Turbo. Run on 22 Nov, 2024, (f) GPT-4o. Run on 22 Nov, 2024.
Fig. 1 Heatmap representing the frequency of extracted information for the three clinical variables (pT, pN, and LVI) across the 10 runs (N = 163) for the six LLMs: Dolly-v2-12b, Vicuna-v1.3-13b, Llama-2.0-13b, Llama-3.3-70b, GPT-4 Turbo, GPT-4o. The input for all the six models was the snippet reports and in their original sequence. The y-axis displays the categories for the specified clinical variables: pT (N/A, invalid, pT0, pTis, pTa, pT1, pT2, pT3, pT4), pN (N/A, invalid, N0, N1, N2, N3), and LVI (N/A, invalid, Yes, No), in which ‘N/A’ indicates that relevant information in the input report for LLMs was unavailable or absent, and ‘invalid’ signifies that the LLM’s output could not be interpreted as meaningful or valid for the given variable. The x-axis corresponds to individual cases. Color intensity indicates the frequency of each category in the data. The color bar on the right provides a reference for the frequency scale. Orange rectangles highlight the ground truth values for each case, enabling comparison with the LLM extractions. (a) Dolly-v2-12b, (b) Vicuna-v1.3-13b, (c) Llama-2.0-13b, (d) Llama-3.3-70b, (e) GPT-4 Turbo. Run on 22 Nov, 2024, (f) GPT-4o. Run on 22 Nov, 2024.CC BY-NC-ND · PubMed Central
Fig. 2 Performance comparison (measured by AUC) of survival prediction across 10 runs from different conditions: (1) different model, (2) GPT-4o across 4 time points, (3) GPT-4 Turbo across snippet or full reports and original or random input sequence, (4) GPT-4o across snippet or full reports and original or random input sequence. The black dots marked the outliers that fall outside of the interquartile range (IQR) which is calculated as \:IQR={Q}_{3}-{Q}_{1}, where \:{Q}_{1} is the first quartile and \:{Q}_{3} is the third quartile. The three horizontal dashed lines represent the AUCs of models based on the ground truth of clinical variables. (a) Six different LLMs: Dolly-v2-12b, Vicuna-v1.3-13b, Llama-2.0-13b, Llama-3.3-70b, GPT-4 Turbo, and GPT-4o, (b) Four time points running GPT-4o. The first three timepoints have an interval of 10 days, and the last interval is 30 days, (c) GPT-4 Turbo tested by different length of reports, (snippet or full reports) and original sequence, (OS) or random sequence, (RS), (d) GPT-4o tested by different length of reports, (snippet or full reports) and original sequence, (OS) or random sequence, (RS)
Fig. 2 Performance comparison (measured by AUC) of survival prediction across 10 runs from different conditions: (1) different model, (2) GPT-4o across 4 time points, (3) GPT-4 Turbo across snippet or full reports and original or random input sequence, (4) GPT-4o across snippet or full reports and original or random input sequence. The black dots marked the outliers that fall outside of the interquartile range (IQR) which is calculated as \:IQR={Q}_{3}-{Q}_{1}, where \:{Q}_{1} is the first quartile and \:{Q}_{3} is the third quartile. The three horizontal dashed lines represent the AUCs of models based on the ground truth of clinical variables. (a) Six different LLMs: Dolly-v2-12b, Vicuna-v1.3-13b, Llama-2.0-13b, Llama-3.3-70b, GPT-4 Turbo, and GPT-4o, (b) Four time points running GPT-4o. The first three timepoints have an interval of 10 days, and the last interval is 30 days, (c) GPT-4 Turbo tested by different length of reports, (snippet or full reports) and original sequence, (OS) or random sequence, (RS), (d) GPT-4o tested by different length of reports, (snippet or full reports) and original sequence, (OS) or random sequence, (RS)CC BY-NC-ND · PubMed Central
Fig. 3 Study design. The 163 EMRs (selected from 781 patients) were input to various LLMs to perform information retrieval tasks. Each report was input 10 times independently to assess the reproducibility of each LLM. The extracted information was input to downstream models to perform survival prediction. One model was a nomogram that was solely based on clinical information and the other was a multi-modal model that was based on clinical information and image features. The accuracy and consistency of extraction and survival prediction were evaluated.
Fig. 3 Study design. The 163 EMRs (selected from 781 patients) were input to various LLMs to perform information retrieval tasks. Each report was input 10 times independently to assess the reproducibility of each LLM. The extracted information was input to downstream models to perform survival prediction. One model was a nomogram that was solely based on clinical information and the other was a multi-modal model that was based on clinical information and image features. The accuracy and consistency of extraction and survival prediction were evaluated.CC BY-NC-ND · PubMed Central
Fig. 4 Prompt used for extracting the clinical variables from the RMRs.
Fig. 4 Prompt used for extracting the clinical variables from the RMRs.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 20 references

Research works printed but not linked: 3

  1. Kim, Y. et al. Mdagents: an adaptive collaboration of Llms for medical decision-making. Adv. Neural. Inf. Process. Syst. 37, 79410–79452 (2025).
  2. Abbasi Yadkori, Y., Kuzborskij, I., György, A. & Szepesvari, C. To believe or not to believe your LLM: iterative prompting for estimating epistemic uncertainty. Adv. Neural. Inf. Process. Syst. 37, 58077–58117 (2025).
  3. Thatoi, P., Choudhary, R., Shiwlani, A., Qureshi, H. A. & Kumar, S. Natural Language processing (NLP) in the extraction of clinical information from electronic health records (EHRs) for cancer prognosis. Int. J. 10, 2676–2694 (2023).

Cited by

See all 9

Metadata

Authors
Sun, D., Hadjiiski, L., Bruno, G., Gormley, J., Chan, H.-P., Caoili, E. M., Cohan, R. H., Alva, A., Mihalcea, R., Zhou, C., & Gulani, V.
Affiliations
University of Michigan (US)
Corresponding author
Sun, Di
Venue
Scientific Reports
Published
2025-11-21
Volume
15
Issue
1
Pages
43773-43773
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4416520295
Cited by
9
References
20
Citations by year
2026: 9
FWCI
3.60
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institutes of Health (U01-CA232931)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
Keywords
large language models, clinical information extraction, bladder cancer, survival prediction, electronic health records, clinical text, LLM reliability, GPT-4, llamas, vicuña, Dolly, Fleiss' kappa, model variability
MeSH terms
Machine Learning, Large Language Models, Aged, Urinary Bladder Neoplasms, Female, Humans, Male, Middle Aged, Natural Language Processing, Prognosis, Reproducibility of Results, Data Mining, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis. https://app.cochat.ai/papers/10.1038/s41598-025-27593-7
The paper
Sun, D., Hadjiiski, L., Bruno, G., Gormley, J., Chan, H.-P., Caoili, E. M., Cohan, R. H., Alva, A., Mihalcea, R., Zhou, C., & Gulani, V. (2025). Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis. Scientific Reports. https://doi.org/10.1038/s41598-025-27593-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.