CoChat
Sign Up
Cited by 247Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsClinical Reasoning and Diagnostic Skills

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

Open access
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., Steyerberg, E. W.·2022·BMC Medical Research Methodology
DOI 10.1186/s12874-022-01801-8PMC PMC9742671
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01801-8

Cites

116 of 116 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most studies evaluate ICU prediction models exclusively against retrospective labels, without systematic comparison to clinician judgment or assessment of clinical significance.

    “Crucially, most studies evaluate models exclusively against retrospective labels, without systematic comparison to clinician judgment or assessment of clinical significance [13].”

    From A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking (2026) · cites this paper as [13]

    The Background says empirical evidence for the impact or usefulness of prediction models remains scarce, which supports the concern about limited assessment of clinical significance. It does not establish that most ICU studies evaluate models exclusively against retrospective labels or omit systematic comparison with clinician judgment; those specifics are not addressed in the passages reviewed. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 117 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. [2]Steyerberg EW. Clinical prediction models. New York: Springer; 2019.
  2. [31]Moorthie S. What is clinical utility?: PHG Foundation - University of Cambridge. [Available from: https://www.phgfoundation.org/explainer/clinical-utility]. Accessed 11 Aug 2022.
  3. [107]Cox DR Snell EJ Analysis of binary data. The concise encyclopedia of statistics 2008 New York Springer New York 4 5

Cited by

Sorted by citations

Related

Authors
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W.
Affiliations
  • Leiden University Medical Center (NL)
  • The Netherlands Cancer Institute (NL)
Corresponding author
Binuya, M. A. E.
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2022-12-12
Volume
22
Issue
1
Pages
316-316
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-022-01801-8
PMC
PMC9742671
OpenAlex
W4311668966
Cited by
246
References
117
Citations by year
2026: 114 · 2025: 58 · 2024: 51 · 2023: 23
FWCI
4.77
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Leiden Repository (Leiden University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Health~Holland (LSHM19121)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
clinical prediction model, model validation, model updating, methodological recommendations, discrimination (statistics), calibration, net benefit, recalibration, model revision, model extension, meta-model, dynamic updating, machine learning models, clinical utility
MeSH terms
Calibration, Humans, Prognosis, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review