Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
We conducted an extensive set of empirical analyses to examine the effect of the number of events per variable (EPV) on the relative performance of three different methods for assessing the predictive accuracy of a logistic regression model: apparent performance in the analysis sample, split-sample validation, and optimism correction using bootstrap methods. Using a single dataset of patients hospitalized with heart failure, we compared the estimates of discriminatory performance from these methods to those for a very large independent validation sample arising from the same population. As anticipated, the apparent performance was optimistically biased, with the degree of optimism diminishing as the number of events per variable increased. Differences between the bootstrap-corrected approach and the use of an independent validation sample were minimal once the number of events per variable was at least 20. Split-sample assessment resulted in too pessimistic and highly uncertain estimates of model performance. Apparent performance estimates had lower mean squared error compared to split-sample estimates, but the lowest mean squared error was obtained by bootstrap-corrected optimism estimates. For bias, variance, and mean squared error of the performance estimates, the penalty incurred by using split-sample validation was equivalent to reducing the sample size by a proportion equivalent to the proportion of the sample that was withheld for model validation. In conclusion, split-sample validation is inefficient and apparent performance is too optimistic for internal validation of regression-based prediction models. Modern validation methods, such as bootstrap-based optimism correction, are preferable. While these findings may be unsurprising to many statisticians, the results of the current study reinforce what should be considered good statistical practice in the development and validation of clinical prediction models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Sample size for estimating calibration depends not on the total sample size but the size of the smaller group, which determines the events per predictor (EPP).65”
From Calibrating machine learning approaches for probability estimation: A comprehensive comparison (2023) · cites this paper as [65]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Harms, especially serious harms, tend to be rare, and such data in nature are more susceptible to random or systematic errors than are common outcomes. 7 8”
From Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study (2022) · cites this paper as [7]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 2×
“A key concept in the development of prediction models for binary outcomes is the number of events per variable (EPV).(8) When outcomes are binary, the number of events is the smaller of the number of subjects who experienced the outcome and the number of subjects who did not experience the outcome.”
BackgroundMethodHarrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Self-citationEstimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,563
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Self-citationLachenbruch, P. A. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess0507 · Cited by 1,310
Validation of Regression Models: Methods and Examples
Snee, R. D. (1977) · Technometrics · doi:10.1080/00401706.1977.10489581 · Cited by 1,283 · cited 1×
“Snee agreed that application of the derived model to new data is the preferred approach to model validation.(2) However, he suggests that this is often impractical or is not feasible.”
MethodLee, D. S., Austin, P. C., Rouleau, J. L., Liu, P. P., Naimark, D., & Tu, J. V. (2003) · JAMA · doi:10.1001/jama.290.19.2581 · Cited by 1,272
Self-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
Boosting Algorithms: Regularization, Prediction and Model Fitting
Bühlmann, P., & Hothorn, T. (2007) · Statistical Science · doi:10.1214/07-sts242 · Cited by 726
BOOSTING ALGORITHMS: REGULARIZATION, PREDICTION AND MODEL FITTINGPreprint
Bühlmann, P., & Hothorn, T. (2007) · Cited by 705
Correcting for Optimistic Prediction in Small Data Sets
Smith, G. C. S., Seaman, S. R., Wood, A. M., Royston, P., & White, I. R. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu140 · Cited by 352 · cited 1×
“when the number of EPV was very low).(18) They found that the ‘leave-one-out’ approach resulted in estimates of the c-statistics that were systematically lower than the true c-statistic.”
ResultAustin, P. C., & Steyerberg, E. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-82 · Cited by 270 · cited 1×
“Regardless of the method used, we would note that, in general, rare characteristics are of interest only if they have a strong effect.(23,24) A third limitation was our focus on the c-statistic for assessing model performance.”
MethodSelf-citationTu, J. V., Donovan, L. R., Lee, D. S., Wang, J. T., Austin, P. C., Alter, D. A., & Ko, D. T. (2009) · JAMA · doi:10.1001/jama.2009.1731 · Cited by 258 · cited 2×
“The Enhanced Feedback for Effective Cardiac Treatment (EFFECT) Study was a study designed to improve the quality of care provided to patients with cardiovascular disease in Ontario, Canada.(11)”
MethodSelf-citationPicard, R. R., & Berk, K. N. (1990) · The American Statistician · doi:10.1080/00031305.1990.10475704 · Cited by 217
Airola, A., Pahikkala, T., Waegeman, W., De Baets, B., & Salakoski, T. (2011) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.11.018 · Cited by 166 · cited 1×
“Two recently discussed methods are the ‘leave-one-out’ and ‘leave-pair-out’ methods described byAirola et al.(16) (the leave-one-out approach dates back to at least 1968, when it was examined by Lachenbruch and Mickey in the context of discriminant analysis(17)).”
MethodAustin, P. C. (2007) · Statistics in Medicine · doi:10.1002/sim.2770 · Cited by 156 · cited 1×
“Prior studies that found that the use of flexible methods to model non-linear relationships between continuous covariates and the log-odds of the outcome resulted in very minor improvements in model accuracy for predicting mortality in cardiovascular patients.(13,14) In the data used for the current analyses, only 120…”
MethodSelf-citationAustin, P. C., Lee, D. S., Steyerberg, E. W., & Tu, J. V. (2012) · Biometrical Journal · doi:10.1002/bimj.201100251 · Cited by 79
Self-citationAustin, P. C., & Steyerberg, E. W. (2013) · Statistics in Medicine · doi:10.1002/sim.5598 · Cited by 47 · cited 1×
“Regardless of the method used, we would note that, in general, rare characteristics are of interest only if they have a strong effect.(23,24) A third limitation was our focus on the c-statistic for assessing model performance.”
MethodSelf-citationValidating Regression Procedures With New Data
Berk, K. N. (1984) · Technometrics · doi:10.1080/00401706.1984.10487985 · Cited by 22 · cited 1×
“Berk suggested that the best way to validate the predictive accuracy of a model is to apply it to new data.(15) Snee agreed that application of the derived model to new data is the preferred approach to model validation.”
Method(1996)
Cited by
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
“We recently confirmed that a split sample approach with 50% held out leads to models with a suboptimal performance, that is, models with unstable and on average the same performance as obtained with half the sample size [11].”
Self-citationExternal validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Metadata
- Authors
- Austin, P. C., & Steyerberg, E. W.
- Affiliations
- University of Toronto (CA)
- Erasmus MC (NL)
- Corresponding author
- Austin, Peter C
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2014-11-19
- Volume
- 26
- Issue
- 2
- Pages
- 796-808
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- PMC
- PMC5394463
- OpenAlex
- W2100697007
- Cited by
- 516
- References
- 31
- Citations by year
- 2026: 104 · 2025: 77 · 2024: 53 · 2023: 66 · 2022: 55 · 2021: 40 · 2020: 40 · 2019: 26 · 2018: 27 · 2017: 10 · 2016: 9 · 2015: 7
- FWCI
- 9.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Statistical Methods in Medical Research (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Funders
- Heart and Stroke Foundation of Canada
- Institute for Clinical Evaluative Sciences
- Ontario Ministry of Health and Long-Term Care
- Canadian Institutes of Health Research (86508, MOP86508)
- National Institute of Neurological Disorders and Stroke (U01NS086294)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- logistic regression, out-of-sample validation, optimism correction, bootstrap validation, split-sample validation, prediction accuracy, heart failure, clinical prediction model, model validation
- MeSH terms
- Cardiovascular Diseases, Data Interpretation, Statistical, Humans, Prognosis, Risk Factors, Reproducibility of Results, Logistic Models, Sample Size, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


