Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study
Open accessAbstract
When developing risk prediction models on datasets with limited sample size, shrinkage methods are recommended. Earlier studies showed that shrinkage results in better predictive performance on average. This simulation study aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome. We compared standard maximum likelihood with the following shrinkage methods: uniform shrinkage (likelihood-based and bootstrap-based), penalized maximum likelihood (ridge) methods, LASSO logistic regression, adaptive LASSO, and Firth’s correction. In the simulation study, we varied the number of predictors and their strength, the correlation between predictors, the event rate of the outcome, and the events per variable. In terms of results, we focused on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). The results can be summarized into three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. In contrast to other shrinkage approaches, Firth’s correction had a small shrinkage effect but showed low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative, with Firth’s correction as the exception. We conclude that, despite improved performance on average, shrinkage often worked poorly in individual datasets, in particular when it was most needed. The results imply that shrinkage methods do not solve problems associated with small sample size or low number of events per variable.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When sample size is simply too small, even penalization methods have been shown ineffective to mitigate overfitting.
“It should be noted that when the sample size is simply too small, even penalization methods have been shown ineffective to mitigate overfitting83,84.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [83]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · see citing works
Ridge regression: Biased estimation for nonorthogonal problems
Hoerl, A. E., & Kennard, R. W. (1970) · Technometrics · doi:10.1080/00401706.1970.10488634 · Cited by 9,648 · see citing works
Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
B, J. F. A., & Li, R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382273 · Cited by 8,919 · see citing works
The adaptive lasso and its oracle properties
Zou, H. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000735 · Cited by 7,470 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Regression and time series model selection in small samples
HURVICH, C. M., & TSAI, C.-L. (1989) · Biometrika · doi:10.1093/biomet/76.2.297 · Cited by 6,586 · see citing works
Bias reduction of maximum likelihood estimates
FIRTH, D. (1993) · Biometrika · doi:10.1093/biomet/80.1.27 · Cited by 5,035 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationA solution to the problem of separation in logistic regression
Heinze, G., & Schemper, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1047 · Cited by 2,711 · see citing works
Babyak, M. A. (2004) · Psychosomatic Medicine · doi:10.1097/01.psy.0000127692.23278.a9 · Cited by 2,061 · see citing works
Regression modelling strategies for improved prognostic prediction
Harrell, F. E., Lee, K. L., Califf, R. M., Pryor, D. B., & Rosati, R. A. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030207 · Cited by 1,804 · see citing works
Ridge Estimators in Logistic Regression
le Cessie, S., & van Houwelingen, H. C. (1992) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347628 · Cited by 1,689 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Self-citationSample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K., de Groot, J. D., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 776 · see citing works
How to develop a more accurate risk prediction model when there are few events
Pavlou, M., Ambler, G., Seaman, S. R., Guttmann, O. P., Elliot, P. M., King, M., & Omar, R. (2015) · BMJ · doi:10.1136/bmj.h3868 · Cited by 668 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
No rationale for 1 variable per 10 events criterion for binary logistic regression analysis
van Smeden, M., de Groot, J. D., Moons, K., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0267-3 · Cited by 563 · see citing works
Prognostic Modeling with Logistic Regression Analysis
Steyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100106 · Cited by 559 · see citing works
Self-citationTwo further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Adequate sample size for developing prediction models is not simply related to events per variable
Ogundimu, E. O., Altman, D. G., & Collins, G. S. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.02.031 · Cited by 546 · see citing works
Interpreting Incremental Value of Markers Added to Risk Prediction Models
Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws207 · Cited by 506 · see citing works
Firth's logistic regression with rare events: accurate effect estimates and predictions?
Puhr, R., Heinze, G., Nold, M., Lusa, L., & Geroldinger, A. (2017) · Statistics in Medicine · doi:10.1002/sim.7273 · Cited by 483 · see citing works
Timmerman, D., Testa, A. C., Bourne, T., Ferrazzi, E. M., Ameye, L., Konstantinovic, M. L., CALSTER, B. V., Collins, W. P., Vergote, I. B., Huffel, S. V., et al. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.01.7632 · Cited by 480 · see citing works
Pavlou, M., Ambler, G., Seaman, S. R., Iorio, M. D., & Omar, R. (2015) · Statistics in Medicine · doi:10.1002/sim.6782 · Cited by 421 · see citing works
Courvoisier, D. S., Combescure, C., Agoritsas, T., Gayet‐Ageron, A., & Perneger, T. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.012 · Cited by 318 · see citing works
Separation in Logistic Regression: Causes, Consequences, and Control
Mansournia, M. A., Geroldinger, A., Greenland, S., & Heinze, G. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx299 · Cited by 302 · see citing works
Schaefer, R. L., Roi, L. D., & Wolfe, R. A. (1984) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928408828664 · Cited by 273 · see citing works
Validation of prediction models based on lasso regression with multiply imputed data
Musoro, J. Z., Zwinderman, A. H., Puhan, M. A., ter Riet, G., & Geskus, R. B. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-116 · Cited by 181 · see citing works
Sample size considerations and predictive performance of multinomial logistic prediction models
de Jong, V. D., Eijkemans, M., van Calster, B., Timmerman, D., Moons, K., Steyerberg, E., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8063 · Cited by 140 · see citing works
Self-citationApplication of Shrinkage Techniques in Logistic Regression Analysis: A Case Study
Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00157 · Cited by 124 · see citing works
Self-citationAn evaluation of penalised survival methods for developing prognostic models with rare events
Ambler, G., Seaman, S. R., & Omar, R. (2011) · Statistics in Medicine · doi:10.1002/sim.4371 · Cited by 108 · see citing works
Strategies for comparing treatments on a binary response with multi-centre data
Agresti, A., & Hartzel, J. S. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8 · Cited by 102 · see citing works
Shrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy
van Houwelingen, H. C. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00154 · Cited by 86 · see citing works
Shrinkage and model selection with correlated variables via weighted fusion
Daye, Z. J., & Jeng, X. J. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.11.007 · Cited by 69 · see citing works
Flagged first, then by citations
Research works printed but not linked: 26
- Steyerberg EW. Clinical prediction models (2 nd edition). New York: Springer, 20109.
- Steyerberg EW, Eijkemans MJ, Harrell FE Jr, et al. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 2000; 19: 1059-1079.
- Steyerberg EW. Clinical prediction models (2 nd edition). New York: Springer, 2019.
- Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 2004; 66: 411-421.
- Harrell FE, Lee KL, Califf RM, et al. Regression modelling strategies for improved prognostic prediction. Stat Med 1984; 3: 143-152.
- Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373-1379.
- Steyerberg EW, Eijkemans MJ, Harrell FE Jr, et al. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 2000; 19: 1059-1079.
- Pavlou M, Ambler G, Seaman SR, et al. How to develop a more accurate risk prediction model when there are few events. BMJ 2015; 351: h3868.
- Courvoisier DS, Combescure C, Agoritsas T, et al. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol 2011; 64: 993-1000.
- van Smeden M, de Groot JA, Moons KG, et al. Sample size for binary logistic prediction models: Beyond events per variable criteria No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol 2016; 16: 163.
- van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Meth Med Res 2019; 28: 2455-2474.
- Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II -binary and time-to-event outcomes. Stat Med 2019; 38: 1276-1296.
- Ogundimu EO, Altman DG and Collins GS. Adequate sample size for developing prediction models is not simply related to events per variable. J Clin Epidemiol 2016; 76: 175-182.
- Steyerberg EW. Eijkemans MJC and Habbema JDF. Application of shrinkage techniques in logistic regression analysis: a case study. Stat Neerl 2001; 55: 76-88.
- Steyerberg EW. Eijkemans MJC, Harrell FE Jr, et al. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 2001; 21: 45-56.
- Ambler G, Seaman S and Omar RZ. An evaluation of penalised survival methods for developing prognostic models with rare events. Stat Med 2012; 31: 1150-1161.
- Pavlou M, Ambler G, Seaman SR, et al. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Stat Med 2016; 35: 1159-1177.
- Puhr R, Heinze G, Nold M, et al. Firth's logistic regression with rare events: accurate effect estimates and predictions? Stat Med 2017; 36: 2302-2317.
- De Jong VMT, Eijkemans MJC, Van Calster B, et al. Sample size considerations and predictive performance of multinomial logistic prediction models. Stat Med 2019; 38: 1601-1619.
- Copas JB. Regression, prediction and shrinkage. J R Stat Soc B 1983; 45: 311-354.
- Firth D. Bias reduction of maximum likelihood estimates. Biometrika 1993; 80: 27- 38.
- van Houwelingen JC. Shrinkage and penalized likelihood as methods to improve predictive accuracy. Stat Neerl 2001; 55: 17-34.
- Hoerl AE and Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 1970; 12: 55-67.
- Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc B 1996; 58: 267-288.
- Musoro JZ, Zwinderman AH, Puhan MA, et al. Validation of prediction models based on lasso regression with multiply imputed data. BMC Med Res Methodol 2014; 14: 116.
- Harrell FE Jr. Regression modeling strategies (2 nd edition). New York: Springer, 2015.
Cited by
Sorted by citations
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Self-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
- Authors
- Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W.
- Affiliations
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands
- Leiden University Medical Center (NL)
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2020-05-13
- Volume
- 29
- Issue
- 11
- Pages
- 3166-3178
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W3024192828
- Cited by
- 120
- References
- 39
- Citations by year
- 2026: 18 · 2025: 19 · 2024: 15 · 2023: 25 · 2022: 17 · 2021: 20 · 2020: 6
- FWCI
- 11.98
- Type
- journal article
- Language
- en
- Open access
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by-nc)
- Lirias (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc)
- Funders
- Fonds Wetenschappelijk Onderzoek (G0B4716N)
- Onderzoeksraad, KU Leuven (C24/15/037)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- risk prediction model, calibration slope, small sample size, ridge regression, LASSO, adaptive LASSO, overfitting, binary outcomes
- MeSH terms
- Prognosis, Regression Analysis, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
