CoChat
Sign Up
Cited by 2,746Statistical Methods in EpidemiologyStatistical Methods and InferenceArtificial Intelligence in Healthcare

Internal validation of predictive models

Steyerberg, Ewout W, Harrell, Frank E, Borsboom, Gerard J.J.M, Eijkemans, M.J.C, Vergouwe, Yvonne, Habbema, J.Dik F·2001·Journal of Clinical Epidemiology
DOI 10.1016/s0895-4356(01)00341-9
Open this paper in CoChat — free account

Abstract

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Integrity signals

Registry

Crossref · 10.1016/s0895-4356(01)00341-9

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The RCT classifier performance was evaluated using 5000 bootstrap iterations, with threshold set on a random sample with replacement and evaluated on out-of-sample data.

    “we conducted 5000 bootstrap iterations, in which a classification threshold was set on a random sample with replacement of the Hedges dataset, and evaluated on the data not included in that sample.19”

    From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [19]

  • Adequate reporting of predictor selection methods used is important, because the number of candidate predictors and how they were selected can influence the specific predictors included in the final multivariable model and affect interpretation of results.

    “Adequate reporting of predictor selection methods used is important, because the number of candidate predictors and how they were selected at various stages of the study can both influence the specific predictors included in the final multivariable model, and thus affect the interpretation of the results 12,13,43,56.”

    From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [43]

  • Selection based on (significant) univariable predictor-outcome associations is not recommended, as this method increases the chance of biased results in terms of spurious predictors and overfitted and unstable models.

    “However, many methodological reports have shown that selection based on (significant) univariable predictor–outcome associations is not recommended, as this method increases the chance of biased results in terms of spurious predictors and overfitted and unstable models [12],[13],[43],[56].”

    From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [43]

  • The extent of optimism in apparent model performance may be estimated with internal validation techniques.

    “The extent of this optimism may be estimated with so-called internal validation techniques [10],[12],[43],[70]”

    From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [43]

  • Bootstrapping generally leads to more stable and less biased results.

    “Bootstrapping generally leads to more stable and less biased results,93 and is therefore recommended for internal validation.”

    From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [93]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F.
Affiliations
  • Erasmus University Rotterdam (NL)
  • University of Virginia (US)
Corresponding author
Steyerberg, Ewout W
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2001-08
Volume
54
Issue
8
Pages
774-781
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/s0895-4356(01)00341-9
OpenAlex
W2074932800
Cited by
2,746
References
32
Citations by year
2026: 285 · 2025: 157 · 2024: 165 · 2023: 168 · 2022: 164 · 2021: 215 · 2020: 173 · 2019: 173 · 2018: 123 · 2017: 131 · 2016: 132 · 2015: 145 · 2014: 120 · 2013: 97 · 2012: 93
FWCI
21.71
Type
journal article
Language
en
Open access
closed
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
Funders
Koninklijke Nederlandse Akademie van Wetenschappen
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
internal validation, prediction model, cross-validation, bootstrapping, split-sample validation, overfitting, 30-day mortality, acute myocardial infarction, logistic regression, calibration, discrimination (statistics)
MeSH terms
Aged, Female, Humans, Male, Myocardial Infarction, Predictive Value of Tests, Reproducibility of Results, Bias, Logistic Models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review