Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models

Open access
Austin, Peter C, Steyerberg, Ewout W · 2017 · Statistical Methods in Medical Research

Abstract

We conducted an extensive set of empirical analyses to examine the effect of the number of events per variable (EPV) on the relative performance of three different methods for assessing the predictive accuracy of a logistic regression model: apparent performance in the analysis sample, split-sample validation, and optimism correction using bootstrap methods. Using a single dataset of patients hospitalized with heart failure, we compared the estimates of discriminatory performance from these methods to those for a very large independent validation sample arising from the same population. As anticipated, the apparent performance was optimistically biased, with the degree of optimism diminishing as the number of events per variable increased. Differences between the bootstrap-corrected approach and the use of an independent validation sample were minimal once the number of events per variable was at least 20. Split-sample assessment resulted in too pessimistic and highly uncertain estimates of model performance. Apparent performance estimates had lower mean squared error compared to split-sample estimates, but the lowest mean squared error was obtained by bootstrap-corrected optimism estimates. For bias, variance, and mean squared error of the performance estimates, the penalty incurred by using split-sample validation was equivalent to reducing the sample size by a proportion equivalent to the proportion of the sample that was withheld for model validation. In conclusion, split-sample validation is inefficient and apparent performance is too optimistic for internal validation of regression-based prediction models. Modern validation methods, such as bootstrap-based optimism correction, are preferable. While these findings may be unsurprising to many statisticians, the results of the current study reinforce what should be considered good statistical practice in the development and validation of clinical prediction models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/0962280214558972

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Mean estimated c−statistic for different validation methods.
Figure 1. Mean estimated c−statistic for different validation methods.CC BY-NC · PubMed Central
Figure 2. Standard deviation of estimated c−statistic for different validation methods.
Figure 2. Standard deviation of estimated c−statistic for different validation methods.CC BY-NC · PubMed Central
Figure 3. Mean squared error (MSE) of different estimation methods.
Figure 3. Mean squared error (MSE) of different estimation methods.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 31 references

Research works printed but not linked: 2

  1. 10.2307/2529937 · doi:10.2307/2529937
  2. Freund Y, 1996, In: Machine learning: proceedings of the thirteenth international conference, 148

Cited by

See all 516

Metadata

Authors
Austin, P. C., & Steyerberg, E. W.
Affiliations
  • University of Toronto (CA)
  • Erasmus MC (NL)
Corresponding author
Austin, Peter C
Venue
Statistical Methods in Medical Research
Published
2014-11-19
Volume
26
Issue
2
Pages
796-808
ISSN
  • 0962-2802 (ISSN-L)
  • 1477-0334
OpenAlex
W2100697007
Cited by
516
References
31
Citations by year
2026: 104 · 2025: 77 · 2024: 53 · 2023: 66 · 2022: 55 · 2021: 40 · 2020: 40 · 2019: 26 · 2018: 27 · 2017: 10 · 2016: 9 · 2015: 7
FWCI
9.16
Type
journal article
Language
en
Open access
Locations
  • Statistical Methods in Medical Research (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
Funders
  • Heart and Stroke Foundation of Canada
  • Institute for Clinical Evaluative Sciences
  • Ontario Ministry of Health and Long-Term Care
  • Canadian Institutes of Health Research (86508, MOP86508)
  • National Institute of Neurological Disorders and Stroke (U01NS086294)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
logistic regression, out-of-sample validation, optimism correction, bootstrap validation, split-sample validation, prediction accuracy, heart failure, clinical prediction model, model validation
MeSH terms
Cardiovascular Diseases, Data Interpretation, Statistical, Humans, Prognosis, Risk Factors, Reproducibility of Results, Logistic Models, Sample Size, Biostatistics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. https://app.cochat.ai/papers/10.1177/0962280214558972
The paper
Austin, P. C., & Steyerberg, E. W. (2017). Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Statistical Methods in Medical Research. https://doi.org/10.1177/0962280214558972

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.