CoChat
Sign Up
Cited by 247Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsClinical Reasoning and Diagnostic Skills

The DOI matches its Crossref record; 116 of the 117 references it cites were found in a registry or an index.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

Open access
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., Steyerberg, E. W.·2022·BMC Medical Research Methodology
DOI 10.1186/s12874-022-01801-8PMC PMC9742671
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01801-8

Cites

116 of 116 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Framework from derivation to implementation of clinical prediction models. The focus of this systematic review is on model evaluation (validation, impact assessment) and updating
Fig. 1 Framework from derivation to implementation of clinical prediction models. The focus of this systematic review is on model evaluation (validation, impact assessment) and updatingCC BY · PubMed Central
Fig. 2 A summary of methodological guidance for model validation. Internal validation is the minimum requirement for clinical prediction models. External validation is recommended to evaluate model generalizability in different but plausibly related settings. Designs for validation studies differ in strength (e.g., temporal validation is a weak form of validation, and data splitting is discouraged). Examination of two validation aspects (discrimination and calibration) is recommended for assessing statistical performance irrespective of the type of validation. Clinical usefulness is a common area between validation and impact assessment, and its examination is advised for assessing the clinical performance of models intended to be used for medical decision-making
Fig. 2 A summary of methodological guidance for model validation. Internal validation is the minimum requirement for clinical prediction models. External validation is recommended to evaluate model generalizability in different but plausibly related settings. Designs for validation studies differ in strength (e.g., temporal validation is a weak form of validation, and data splitting is discouraged). Examination of two validation aspects (discrimination and calibration) is recommended for assessing statistical performance irrespective of the type of validation. Clinical usefulness is a common area between validation and impact assessment, and its examination is advised for assessing the clinical performance of models intended to be used for medical decision-makingCC BY · PubMed Central
Fig. 3 A summary of methodological guidance for the assessment of a model’s impact. Potential impact can be examined through clinical performance measures (e.g., Net Benefit, Decision Curve Analysis) or health economic analysis (e.g., decision-analytic models). Assessing actual impact requires comparative empirical studies, such as cluster randomized trials or other designs (e.g., stepped-wedge trials, before-after studies)
Fig. 3 A summary of methodological guidance for the assessment of a model’s impact. Potential impact can be examined through clinical performance measures (e.g., Net Benefit, Decision Curve Analysis) or health economic analysis (e.g., decision-analytic models). Assessing actual impact requires comparative empirical studies, such as cluster randomized trials or other designs (e.g., stepped-wedge trials, before-after studies)CC BY · PubMed Central
Fig. 4 A summary of methodological guidance for model updating. Simple updating (e.g., recalibration) is often sufficient when the differences between the derivation and new data are minimal. When the differences are more substantial, more extensive updating (e.g., partial to full revision) may be appropriate. Model extension allows the incorporation of new markers in a model. Multiple models may be combined, for instance, using meta-analytical approaches, to develop a meta-model that can be further updated for a new dataset. Updating can also be done periodically or continuously, resulting in dynamically updated models
Fig. 4 A summary of methodological guidance for model updating. Simple updating (e.g., recalibration) is often sufficient when the differences between the derivation and new data are minimal. When the differences are more substantial, more extensive updating (e.g., partial to full revision) may be appropriate. Model extension allows the incorporation of new markers in a model. Multiple models may be combined, for instance, using meta-analytical approaches, to develop a meta-model that can be further updated for a new dataset. Updating can also be done periodically or continuously, resulting in dynamically updated modelsCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 117 references

Research works printed but not linked: 2

  1. Moorthie S. What is clinical utility?: PHG Foundation - University of Cambridge. [Available from: https://www.phgfoundation.org/explainer/clinical-utility]. Accessed 11 Aug 2022.
  2. Cox DR, Snell EJ. Analysis of binary data. The concise encyclopedia of statistics. New York: Springer New York; 2008. p. 4–5.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W.
Affiliations
  • Leiden University Medical Center (NL)
  • The Netherlands Cancer Institute (NL)
Corresponding author
Binuya, M. A. E.
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2022-12-12
Volume
22
Issue
1
Pages
316-316
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-022-01801-8
PMC
PMC9742671
OpenAlex
W4311668966
Cited by
246
References
117
Citations by year
2026: 114 · 2025: 58 · 2024: 51 · 2023: 23
FWCI
4.77
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Leiden Repository (Leiden University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Health~Holland (LSHM19121)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
clinical prediction model, model validation, model updating, methodological recommendations, discrimination (statistics), calibration, net benefit, recalibration, model revision, model extension, meta-model, dynamic updating, machine learning models, clinical utility
MeSH terms
Calibration, Humans, Prognosis, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review