Cited by 141Statistical Methods in EpidemiologyArtificial Intelligence in HealthcareMachine Learning in Healthcare

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

Open access
Dhiman, Paula, Ma, Jie, Andaur Navarro, Constanza L., Speich, Benjamin, Bullock, Garrett, Damen, Johanna A. A., Hooft, Lotty, Kirtley, Shona +4 more·2022·BMC Medical Research Methodology

Abstract

BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01577-x

Cites

125 of 125 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

1 of the 126 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., Moons, K. G. M., & Collins, G. S.
Affiliations
  • University of Oxford (GB)
  • University of Basel (CH)
  • Utrecht University (NL)
  • Keele University (GB)
Corresponding author
Dhiman, Paula
Venue
BMC Medical Research Methodology
Published
2022-04-08
Volume
22
Issue
1
Pages
101-101
ISSN
1471-2288 (ISSN-L)
OpenAlex
W4223540626
Cited by
140
References
126
Citations by year
2026: 37 · 2025: 34 · 2024: 42 · 2023: 23 · 2022: 4
FWCI
3.72
Type
journal article
Language
en
Open access
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
Funders
  • Cancer Research UK (27294)
  • National Institute for Health and Care Research
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
prognostic model, machine learning, oncology, methodological quality, tripod, PROBAST, charm, regression models, ensemble learning, sample size justification, predictor selection, time-to-event outcomes, internal validation, calibration, cancer prognosis, cancer survival prediction, cancer risk prediction, tumor response prediction, tumor recurrence
MeSH terms
Machine Learning, Humans, Medical Oncology, Prognosis, Research Design, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review