The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Cross validation for model selection: A review with examples from ecology
Open accessAbstract
Specifying, assessing, and selecting among candidate statistical models is fundamental to ecological research. Commonly used approaches to model selection are based on predictive scores and include information criteria such as Akaike's information criterion, and cross validation. Based on data splitting, cross validation is particularly versatile because it can be used even when it is not possible to derive a likelihood (e.g., many forms of machine learning) or count parameters precisely (e.g., mixed‐effects models). However, much of the literature on cross validation is technical and spread across statistical journals, making it difficult for ecological analysts to assess and choose among the wide range of options. Here we provide a comprehensive, accessible review that explains important—but often overlooked—technical aspects of cross validation for model selection, such as: bias correction, estimation uncertainty, choice of scores, and selection rules to mitigate overfitting. We synthesize the relevant statistical advances to make recommendations for the choice of cross‐validation technique and we present two ecological case studies to illustrate their application. In most instances, we recommend using exact or approximate leave‐one‐out cross validation to minimize bias, or otherwise k‐fold with bias correction if k < 10. To mitigate overfitting when using cross validation, we recommend calibrated selection via our recently introduced modified one‐standard‐error rule. We advocate for the use of predictive scores in model selection across a range of typical modeling goals, such as exploration, hypothesis testing, and prediction, provided that models are specified in accordance with the stated goal. We also emphasize, as others have done, that inference on parameter estimates is biased if preceded by model selection and instead requires a carefully specified single model or further technical adjustments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Generalized Additive Models: An Introduction with R
Wood, S. N. (2017) · doi:10.1201/9781315370279 · Cited by 13,852
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Dormann, C. F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G., Marquéz, J. R. G., Gruber, B., Lafourcade, B., Leitão, P. J., et al. (2013) · Ecography · doi:10.1111/j.1600-0587.2012.07348.x · Cited by 9,998
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399
ALLOUCHE, O., TSOAR, A., & KADMON, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,172
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118
Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., et al. (2017) · Ecography · doi:10.1111/ecog.02881 · Cited by 3,689
A survey of cross-validation procedures for model selectionPreprint
Arlot, S., & Célisse, A. (2010) · Cited by 3,625
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Craven, P., & Wahba, G. (1978) · Numerische Mathematik · doi:10.1007/bf01404567 · Cited by 3,388
Selecting thresholds of occurrence in the prediction of species distributions
Liu, C., Berry, P. M., Dawson, T. P., & Pearson, R. G. (2005) · Ecography · doi:10.1111/j.0906-7590.2005.03957.x · Cited by 2,701
MCMC Using Hamiltonian Dynamics
Neal, R. M. (2011) · Handbook of Markov Chain Monte Carlo · doi:10.1201/b10905-6 · Cited by 2,574
Statistical Learning with Sparsity: The Lasso and Generalizations
Hastie, T., Tibshirani, R., & Wainwright, M. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b18401 · Cited by 2,520
Shmueli, G. (2010) · Statistical Science · doi:10.1214/10-sts330 · Cited by 2,517
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
caret: Classification and Regression Training
Kuhn, M. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.caret · Cited by 1,786
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Predictive Analytics in Information Systems Research1
Shmueli, G., & Koppius, O. R. (2011) · MIS Quarterly · doi:10.2307/23042796 · Cited by 1,328
Making and Evaluating Point Forecasts
Gneiting, T. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.r10138 · Cited by 1,154
Luque, A., Carrasco, A., Martín, A., & de las Heras, A. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.02.023 · Cited by 1,128
Visualization in Bayesian Workflow
Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., & Gelman, A. (2019) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12378 · Cited by 1,122
The Insignificance of Statistical Significance Testing
Johnson, D. H. (1999) · The Journal of Wildlife Management · doi:10.2307/3802789 · Cited by 1,104
A guide to Bayesian model selection for ecologists
Hooten, M. B., & Hobbs, N. T. (2015) · Ecological Monographs · doi:10.1890/14-0661.1 · Cited by 891
Ellison, A. M. (2004) · Ecology Letters · doi:10.1111/j.1461-0248.2004.00603.x · Cited by 786
An Introduction to Causal Inference
Pearl, J. (2010) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1203 · Cited by 755
Frequentist Model Average Estimators
Hjort, N. L., & Claeskens, G. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000000828 · Cited by 688
Submodel Selection and Evaluation in Regression. The X-Random Case
Breiman, L., & Spector, P. (1992) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403680 · Cited by 669
BURMAN, P. (1989) · Biometrika · doi:10.1093/biomet/76.3.503 · Cited by 634
Model Selection and Model Averaging
Claeskens, G., & Hjort, N. L. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511790485 · Cited by 629
Model averaging and muddled multimodel inferences
Cade, B. S. (2015) · Ecology · doi:10.1890/14-1639.1 · Cited by 622
Valid post-selection inference
Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013) · The Annals of Statistics · doi:10.1214/12-aos1077 · Cited by 599
Expected Information as Expected Utility
Bernardo, J. M. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344689 · Cited by 568
rstanarm: Bayesian Applied Regression Modeling via Stan
Gabry, J., & Goodrich, B. (2016) · CRAN: Contributed Packages · doi:10.32614/cran.package.rstanarm · Cited by 503
Exact post-selection inference, with application to the lasso
Lee, J. D., Sun, D. L., Sun, Y., & Taylor, J. E. (2016) · The Annals of Statistics · doi:10.1214/15-aos1371 · Cited by 501
A practical guide to selecting models for exploration, inference, and prediction in ecology
Tredennick, A. T., Hooker, G., Ellner, S. P., & Adler, P. B. (2021) · Ecology · doi:10.1002/ecy.3336 · Cited by 474
Sparsity information and regularization in the horseshoe and other shrinkage priors
Piironen, J., & Vehtari, A. (2017) · Electronic Journal of Statistics · doi:10.1214/17-ejs1337si · Cited by 457
Chicco, D., Warrens, M. J., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 444
Dormann, C. F., Calabrese, J. M., Guillera‐Arroita, G., Matechou, E., Bahn, V., Bartoń, K., Beale, C. M., Ciuti, S., Elith, J., Gerstner, K., et al. (2018) · Ecological Monographs · doi:10.1002/ecm.1309 · Cited by 400
Exact Post-Selection Inference for Sequential Regression Procedures
Tibshirani, R. J., Taylor, J., Lockhart, R., & Tibshirani, R. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1108848 · Cited by 344
Comparison of Bayesian predictive methods for model selection
Piironen, J., & Vehtari, A. (2017) · Statistics and Computing · doi:10.1007/s11222-016-9649-y · Cited by 294
Bayesian Models: A Statistical Primer for Ecologists
Hobbs, N. T., & Hooten, M. B. (2015) · Princeton University Press eBooks · doi:10.1515/9781400866557 · Cited by 249
Shrinkage priors for Bayesian penalized regression
van Erp, S., Oberski, D. L., & Mulder, J. (2019) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2018.12.004 · Cited by 223
Tjørve, E., & Tjørve, K. M. C. (2010) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2010.09.008 · Cited by 206
A cross-validatory method for dependent data
BURMAN, P., CHOW, E., & NOLAN, D. (1994) · Biometrika · doi:10.1093/biomet/81.2.351 · Cited by 202
Predictive models aren't for causal inference
Arif, S., & MacNeil, M. A. (2022) · Ecology Letters · doi:10.1111/ele.14033 · Cited by 167
Spatial Ecology and Conservation Modeling: Applications with R
Fletcher, R., & Fortin, M.-J. (2018) · doi:10.1007/978-3-030-01989-1 · Cited by 155
Fang, Y. (2021) · Journal of Data Science · doi:10.6339/jds.201101_09(1).0002 · Cited by 147
Approximate leave-future-out cross-validation for Bayesian time series models
Bürkner, P., Gabry, J., & Vehtari, A. (2020) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2020.1783262 · Cited by 133
Projective inference in high-dimensional problems: Prediction and feature selection
Piironen, J., Paasiniemi, M., & Vehtari, A. (2020) · Electronic Journal of Statistics · doi:10.1214/20-ejs1711 · Cited by 128
Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods
Merkle, E. C., Furr, D. C., & Rabe‐Hesketh, S. (2019) · Psychometrika · doi:10.1007/s11336-019-09679-0 · Cited by 120
Considerations for assessing model averaging of regression coefficients
Banner, K. M., & Higgs, M. D. (2016) · Ecological Applications · doi:10.1002/eap.1419 · Cited by 119
Nearest neighbour distance matching Leave‐One‐Out Cross‐Validation for map validation
Milà, C., Mateu, J., Pebesma, E., & Meyer, H. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13851 · Cited by 119
Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model ComparisonPreprint
Sivula, T., Magnusson, M., Matamoros, A. A., & Vehtari, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10296 · Cited by 87
Limitations of “Limitations of Bayesian Leave-one-out Cross-Validation for Model Selection”
Vehtari, A., Simpson, D., Yao, Y., & Gelman, A. (2018) · Computational Brain & Behavior · doi:10.1007/s42113-018-0020-6 · Cited by 85
Linear model selection by cross-validation
Rao, C. R., & Wu, Y. (2003) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2003.10.004 · Cited by 61
Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
Bürkner, P., Gabry, J., & Vehtari, A. (2020) · Computational Statistics · doi:10.1007/s00180-020-01045-4 · Cited by 61
Inference After Model Selection
Shen, X., Huang, H., & Ye, J. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000001097 · Cited by 58
Choice of V for V-Fold Cross-Validation in Least-Squares Density EstimationPreprint
Arlot, S., & Lerasle, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1210.5830 · Cited by 57
Selective Inference: The Silent Killer of Replicability
Benjamini, Y. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.fc62b261 · Cited by 54
V-fold cross-validation improved: V-fold penalizationPreprint
Arlot, S. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0802.0566 · Cited by 51
The Coverage Properties of Confidence Regions After Model Selection
Kabaila, P. V. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00089.x · Cited by 42
Asymptotic post-selection inference for the Akaike information criterion
Charkhi, A., & Claeskens, G. (2018) · Biometrika · doi:10.1093/biomet/asy018 · Cited by 38
Post-model-selection inference in linear regression models: An integrated review
Zhang, D., Khalili, A., & Asgharian, M. (2022) · Statistics Surveys · doi:10.1214/22-ss135 · Cited by 37
Model‐Averaged Confidence Intervals
Kabaila, P. V., Welsh, A. H., & Abeysekera, W. (2015) · Scandinavian Journal of Statistics · doi:10.1111/sjos.12163 · Cited by 35
The invasive azedarach scale Aulacaspis crawii (Cockerell, 1898) (Hemiptera, Diaspididae) in India
Joshi, S., Prathapan, K., Sreejith, M., Bhaskar, H., Shukla, S., Kumari, B., & Kunhamu, T. (2022) · NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) · Cited by 31
Parsimonious model selection using information theory: a modified selection rule
Yates, L. A., Richards, S. A., & Brook, B. W. (2021) · Ecology · doi:10.1002/ecy.3475 · Cited by 30
Self-citationSELECTION OF WEIGHTS FOR WEIGHTED MODEL AVERAGING
Garthwaite, P. H., & Mubwandarikwa, E. (2010) · Australian & New Zealand Journal of Statistics · doi:10.1111/j.1467-842x.2010.00589.x · Cited by 29
Metadata
- Authors
- Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W.
- Venue
- Ecological Monographs
- Publisher
- Wiley
- Published
- 2023-01-03
- Volume
- 93
- Issue
- 1
- ISSN
- 0012-9615 (ISSN-L)
- 1557-7015
- OpenAlex
- W4309048734
- Cited by
- 459
- References
- 83
- FWCI
- 52.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.