CoChat
Sign Up
Cited by 814Statistical Methods in EpidemiologyArtificial Intelligence in HealthcareMeta-analysis and systematic reviews

The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.

Evaluation of clinical prediction models (part 1): from development to external validation

Open access
Collins, Gary S, Dhiman, Paula, Ma, Jie, Schlussel, Michael M, Archer, Lucinda, Van Calster, Ben, Harrell, Frank E, Martin, Glen P, Moons, Karel G M, van Smeden, Maarten, Sperrin, Matthew, Bullock, Garrett S, Riley, Richard D·2024·BMJ
DOI 10.1136/bmj-2023-074819PMC PMC10772854
Open this paper in CoChat — free accountOpen PDF

Abstract

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmj-2023-074819

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Different study design and approaches to develop and evaluate the performance of a multivariable prediction model (D=development; V=validation (evaluation)). Adapted from Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594.17 *A study can include more than one analysis type
Fig 1 Different study design and approaches to develop and evaluate the performance of a multivariable prediction model (D=development; V=validation (evaluation)). Adapted from Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594.17 *A study can include more than one analysis typeCC BY · PubMed Central
Fig 2 Variability and overestimation of apparent performance compared to large sample performance, for a model to predict in-hospital mortality within 28 days of trauma injury with increasing sample size of the model development study. ĉ denotes the apparent performance estimate and clarge denotes the performance of the model in the entire CRASH-2 population (n=20 207).22 Red lines=mean ĉ−clarge for each sample size. Jitter has been added to aid display. ĉ−clarge=0 implies no systematic overestimation or underestimation of ĉ
Fig 2 Variability and overestimation of apparent performance compared to large sample performance, for a model to predict in-hospital mortality within 28 days of trauma injury with increasing sample size of the model development study. ĉ denotes the apparent performance estimate and clarge denotes the performance of the model in the entire CRASH-2 population (n=20 207).22 Red lines=mean ĉ−clarge for each sample size. Jitter has been added to aid display. ĉ−clarge=0 implies no systematic overestimation or underestimation of ĉCC BY · PubMed Central
Fig 3 Variability and overestimation of the apparent and internal (split sample and bootstrap) validation performance compared with the large sample performance, for a model to predict in-hospital mortality within 28 days of trauma injury with increasing sample size of the model development study. ĉ denotes the apparent performance estimate and clarge denotes the performance of the model in the entire CRASH-2 population (n=20 207). The red lines denote the mean ĉ−clarge for each sample size and for each approach. Jitter has been added to aid display. Split sample (apparent, 70%)=70% of the available data were used to develop the model, and its (apparent) performance evaluated in this same data. Split sample (validation, 30%)=the performance of the model (developed in 70% of the available data) in the remaining 30% of the data. ĉ−clarge=0 implies no systematic overestimation or underestimation of ĉ
Fig 3 Variability and overestimation of the apparent and internal (split sample and bootstrap) validation performance compared with the large sample performance, for a model to predict in-hospital mortality within 28 days of trauma injury with increasing sample size of the model development study. ĉ denotes the apparent performance estimate and clarge denotes the performance of the model in the entire CRASH-2 population (n=20 207). The red lines denote the mean ĉ−clarge for each sample size and for each approach. Jitter has been added to aid display. Split sample (apparent, 70%)=70% of the available data were used to develop the model, and its (apparent) performance evaluated in this same data. Split sample (validation, 30%)=the performance of the model (developed in 70% of the available data) in the remaining 30% of the data. ĉ−clarge=0 implies no systematic overestimation or underestimation of ĉCC BY · PubMed Central
Fig 4 Graphical illustration of k-fold cross validation. Non-shaded parts used for model development; shaded part used for testing
Fig 4 Graphical illustration of k-fold cross validation. Non-shaded parts used for model development; shaded part used for testingCC BY · PubMed Central
Fig 5 Graphical illustration of internal-external cross validation. Non-shaded parts used for model development; shaded part used for testing
Fig 5 Graphical illustration of internal-external cross validation. Non-shaded parts used for model development; shaded part used for testingCC BY · PubMed Central
Fig 6 Internal-external cross validation of the ISARIC (International Severe Acute Respiratory and Emerging Infection Consortium) 4C model. Adapted from Gupta et al.2 Estimates and confidence intervals taken from original paper where they were reported to two decimal places.
Fig 6 Internal-external cross validation of the ISARIC (International Severe Acute Respiratory and Emerging Infection Consortium) 4C model. Adapted from Gupta et al.2 Estimates and confidence intervals taken from original paper where they were reported to two decimal places.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. 10.1136/bmj.m1328 · DOI · Paper page
  2. 10.1177/962280218784726 · DOI

Cited by

Sorted by citations

See all 814

Related

See all

Metadata

Authors
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., Sperrin, M., Bullock, G. S., & Riley, R. D.
Affiliations
  • Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
  • University of Oxford (GB)
  • National Institute for Health and Care Research (GB)
  • KU Leuven, Department of Development and Regeneration, Leuven, Belgium
  • Vanderbilt University (US)
  • Manchester Academic Health Science Centre (GB)
  • Utrecht University (NL)
  • Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
ORCID
Venue
BMJ
Publisher
BMJ
Published
2024-01-08
Volume
384
Pages
e074819
ISSN
1756-1833 (ISSN-L)
DOI
10.1136/bmj-2023-074819
PMC
PMC10772854
OpenAlex
W4390674087
Cited by
798
References
65
Citations by year
2026: 464 · 2025: 253 · 2024: 81
FWCI
274.95
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • Research Explorer (The University of Manchester) (publishedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, cc-by)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • HighWire Press Open Archive (submittedVersion)
Funders
  • Cancer Research UK
  • KU Leuven
  • Engineering and Physical Sciences Research Council
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
clinical prediction model, model evaluation, external validation, internal validation, prediction accuracy, model heterogeneity, fairness, generalizability
MeSH terms
Humans, Prognosis, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review