Regression, Prediction and Shrinkage
Abstract
Summary The fit of a regression predictor to new data is nearly always worse than its fit to the original data. Anticipating this shrinkage leads to Stein-type predictors which, under certain assumptions, give a uniformly lower prediction mean squared error than least squares. Shrinkage can be particularly marked when stepwise fitting is used: the shrinkage is then closer to that expected of the full regression rather than of the subset regression actually fitted. Preshrunk predictors for selected subsets are proposed and tested on a number of practical examples. Both multiple and binary (logistic) regression models are considered.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Almost 30 years of empirical Bayes research show that modeling approaches that shrink a coefficient part of the way back to zero can dominate all-or-nothing methods.
“by now we have almost 30 years of good empirical Bayes research (e.g., Sclove, Morris, Radhakrishna, 1972; Copas, 1983) to show that modeling approaches that shrink a given coefficient part of the way back to zero can dominate all-or-nothing methods.”
From Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors (1999) · cites this paper as Copas, 1983
Cited by
Sorted by citations
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Detecting gene–gene interactions that underlie human diseases
Cordell, H. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2579 · Cited by 1,350 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738 · see citing works
Predictive Analytics in Information Systems Research
Shmueli, G., & Koppius, O. (2010) · doi:10.2139/ssrn.1606674 · Cited by 676 · see citing works
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Guo, S., Popp, J., & Bocklitz, T. (2021) · Nature Protocols · doi:10.1038/s41596-021-00620-3 · Cited by 361 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915 · see citing works
Knee Extension Strength Cutpoints for Maintaining Mobility
Manini, T. M., Visser, M., Won‐Park, S., Patel, K. V., Strotmeyer, E. S., Chen, H., Goodpaster, B., De Rekeneire, N., Newman, A. B., Simonsick, E. M., et al. (2007) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2007.01087.x · Cited by 249 · see citing works
Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
King, M., Walker, C., Levy, G., Bottomley, C., Royston, P., Weich, S., Bellón-Saameño, J. Á., Moreno, B., Švab, I., Rotar, D., et al. (2008) · Archives of General Psychiatry · doi:10.1001/archpsyc.65.12.1368 · Cited by 221 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
Application of Shrinkage Techniques in Logistic Regression Analysis: A Case Study
Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00157 · Cited by 124 · see citing works
Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120 · see citing works
The growing ubiquity of algorithms in society: implications, impacts and innovations
Olhede, S. C., & Wolfe, P. J. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0364 · Cited by 120 · see citing works
Goeminne, L. E., Gevaert, K., & Clement, L. (2016) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m115.055897 · Cited by 110 · see citing works
Analyzing Raman spectroscopic data
Ryabchykov, O., Guo, S., & Bocklitz, T. (2019) · Physical Sciences Reviews · doi:10.1515/psr-2017-0043 · Cited by 74 · see citing works
Applying novel technologies and methods to inform the ontology of self-regulation
Eisenberg, I. W., Bissett, P. G., Canning, J. R., Dallery, J., Enkavi, A. Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A. I., Greene, M. A., Kiernan, M., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.014 · Cited by 61 · see citing works
Validation of equations used to predict plasma osmolality in a healthy adult cohort
Heavens, K. R., Kenefick, R. W., Caruso, E. M., Spitz, M. G., & Cheuvront, S. N. (2014) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.114.091009 · Cited by 60 · see citing works
Jinks, R. C., Royston, P., & Parmar, M. K. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0078-y · Cited by 32 · see citing works
Covariate‐adjusted measures of discrimination for survival data
White, I. R., Rapsomaniki, E., & for the Emerging Risk Factors Collaboration (2015) · Biometrical Journal · doi:10.1002/bimj.201400061 · Cited by 13 · see citing works
- Authors
- Copas, J. B.
- Affiliations
- University of Birmingham (GB)
- Corresponding author
- Copas, J. B.
- ORCID
- Copas, J. B.
- Venue
- Journal of the Royal Statistical Society Series B: Statistical Methodology
- Publisher
- Oxford University Press (OUP)
- Published
- 1983-07-01
- Volume
- 45
- Issue
- 3
- Pages
- 311-335
- ISSN
- 1369-7412 (ISSN-L)
- 1467-9868
- OpenAlex
- W48332454
- Cited by
- 612
- References
- 0
- Citations by year
- 2026: 15 · 2025: 5 · 2024: 16 · 2023: 11 · 2022: 13 · 2021: 39 · 2020: 18 · 2019: 19 · 2018: 27 · 2017: 22 · 2016: 22 · 2015: 23 · 2014: 28 · 2013: 29 · 2012: 25
- FWCI
- 16.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the Royal Statistical Society Series B (Statistical Methodology) (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- shrinkage, mean squared prediction error, stepwise regression, multiple linear regression, logistic regression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
