The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Comparison of Bayesian predictive methods for model selection
Open accessAbstract
The goal of this paper is to compare several widely used Bayesian model selection methods in practical model selection problems, highlight their differences and give recommendations about the preferred approaches. We focus on the variable subset selection for regression and classification and perform several numerical experiments using both simulated and real world data. The results show that the optimization of a utility estimate such as the cross-validation (CV) score is liable to finding overfitted models due to relatively high variance in the utility estimates when the data is scarce. This can also lead to substantial selection induced bias and optimism in the performance evaluation for the selected model. From a predictive viewpoint, best results are obtained by accounting for model uncertainty by forming the full encompassing model, such as the Bayesian model averaging solution over the candidate models. If the encompassing model is too complex, it can be robustly simplified by the projection method, in which the information of the full model is projected onto the submodels. This approach is substantially less prone to overfitting than selection based on CV-score. Overall, the projection method appears to outperform also the maximum a posteriori model and the selection of the most probable variables. The study also demonstrates that the model selection can greatly benefit from using cross-validation outside the searching process both for guiding the model size selection and assessing the predictive performance of the finally selected model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Reversible jump Markov chain Monte Carlo computation and Bayesian model determination
GREEN, P. J. (1995) · Biometrika · doi:10.1093/biomet/82.4.711 · Cited by 5,930
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
Self-citationHoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118 · cited 4×
“Our results agree with what is known about the good performance of the BMA (Hoeting et al., 1999; Raftery and Zheng, 2003).”
MethodResultInfluentialVariable Selection via Gibbs Sampling
George, E. I., & McCulloch, R. E. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476353 · Cited by 2,828
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“However, the topic has been discussed for example by Rencher and Pun (1980), Ambroise and McLachlan (2002), Reunanen (2003), Varma and Simon (2006), and Cawley and Talbot (2010).”
BackgroundWatanabe, S. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1004.2316 · Cited by 2,226
Bernardo, J. M., & Smith, A. F. M. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316870 · Cited by 2,178
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
Bayesian Model Averaging for Linear Regression Models
Raftery, A. E., Hoeting, J. A., & Madigan, D. (1997) · Cited by 1,746
Selection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,480
Bayesian Variable Selection in Linear Regression
Mitchell, T. J., & Beauchamp, J. J. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478694 · Cited by 1,424
The horseshoe estimator for sparse signals
Carvalho, C. M., Polson, N. G., & Scott, J. G. (2010) · Biometrika · doi:10.1093/biomet/asq017 · Cited by 1,381
(1997) · Cited by 1,193
Rubin, D. B. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345338 · Cited by 1,153
A Predictive Approach to Model Selection
Geisser, S., & Eddy, W. F. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481632 · Cited by 988
Good, I. J. (1952) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1952.tb00104.x · Cited by 976
Optimal predictive model selection
Barbieri, M. M., & Berger, J. O. (2004) · The Annals of Statistics · doi:10.1214/009053604000000238 · Cited by 812
A review of Bayesian variable selection methods: what, how and which
O'Hara, R. B., & Sillanpää, M. J. (2009) · Bayesian Analysis · doi:10.1214/09-ba403 · Cited by 771
Model choice: a minimum posterior predictive loss approach
Gelfand, A. (1998) · Biometrika · doi:10.1093/biomet/85.1.1 · Cited by 724
Dimensionality Reduction for Supervised Learning With Reproducing Kernel Hilbert Spaces
Fukumizu, K., Bach, F. R., & Jordan, M. I. (2003) · doi:10.21236/ada446572 · Cited by 723
(1) OPTIMAL PREDICTIVE MODEL SELECTION 1
Barbieri, M. M., Sapienza, U. D. R. “., & Berger, J. O. (2002) · Cited by 640
BURMAN, P. (1989) · Biometrika · doi:10.1093/biomet/76.3.503 · Cited by 634
Multivariate Bayesian Variable Selection and Prediction
Brown, P. J., Vannucci, M., & Fearn, T. (1998) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00144 · Cited by 369
Gene selection: a Bayesian variable selection approach
Lee, K. E., Sha, N., Dougherty, E. R., Vannucci, M., & Mallick, B. K. (2003) · Bioinformatics · doi:10.1093/bioinformatics/19.1.90 · Cited by 352
Laud, P. W., & Ibrahim, J. G. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02028.x · Cited by 350
A survey of Bayesian predictive methods for model assessment, selection and comparison
Vehtari, A., & Ojanen, J. (2012) · Statistics Surveys · doi:10.1214/12-ss102 · Cited by 348
Self-citationBayesian Model Averaging for Linear Regression Models
Raftery, A. E., Madigan, D., & Hoeting, J. A. (1997) · Journal of the American Statistical Association · doi:10.2307/2291462 · Cited by 294
Markov Chain Monte Carlo Methods for Computing Bayes Factors: A Comparative Review
Han, C., & Carlin, B. P. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753208780 · Cited by 275
Discussion: Performance of Bayesian Model Averaging
Raftery, A. E., & Zheng, Y. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000000891 · Cited by 252
The Choice of Variables in Multiple Regression
Lindley, D. V. (1968) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1968.tb01505.x · Cited by 232
Bayesian variable selection with shrinking and diffusing priors
Narisetty, N. N., & He, X. (2014) · The Annals of Statistics · doi:10.1214/14-aos1207 · Cited by 222
Inflation of R 2 in Best Subset Regression
Rencher, A. C., & Pun, F. C. (1980) · Technometrics · doi:10.1080/00401706.1980.10486100 · Cited by 211
Overfitting in making comparisons between variable selection methods
ReunanenJuha (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944978 · Cited by 205
Algebraic Geometry and Statistical Learning Theory
Watanabe, S. (2009) · doi:10.1017/cbo9780511800474 · Cited by 183
Decoupling Shrinkage and Selection in Bayesian Linear Models: A Posterior Summary Perspective
Hahn, P. R., & Carvalho, C. M. (2015) · Journal of the American Statistical Association · doi:10.1080/01621459.2014.993077 · Cited by 178
Bayes Model Averaging with Selection of Regressors
Brown, P. J., Vannucci, M., & Fearn, T. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00348 · Cited by 165
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151
Asymptotic Properties of Bayes Risk for the Horseshoe Prior
Datta, J., & Ghosh, J. K. (2013) · Bayesian Analysis · doi:10.1214/13-ba805 · Cited by 123
A Predictive Model Selection Criterion
Martini, A. S., & Spezzaferri, F. (1984) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1984.tb01302.x · Cited by 96
A Predictive Approach to the Analysis of Designed Experiments
Ibrahim, J. G., & Laud, P. W. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476472 · Cited by 95
Model choice in generalised linear models: a Bayesian approach via Kullback-Leibler projections
Goutis, C. (1998) · Biometrika · doi:10.1093/biomet/85.1.29 · Cited by 94
Brown, P. J. (1999) · Biometrika · doi:10.1093/biomet/86.3.635 · Cited by 75
Variable selection in qualitative models via an entropic explanatory power
Dupuis, J. A., & Robert, C. P. (2002) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(02)00286-0 · Cited by 63
Erratum to: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Vehtari, A., Gelman, A., & Gabry, J. (2017) · Statistics and Computing · doi:10.1007/s11222-016-9709-3 · Cited by 62
Self-citationPeltola, T., Havulinna, A. S., Salomaa, V. V., & Vehtari, A. (2014) · Cited by 25
Self-citationBayesian projection approaches to variable selection in generalized linear models
Nott, D. J., & Leng, C. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.01.036 · Cited by 24
A Bayesian Approach to Selecting Covariates for Prediction
Marriott, J. M., Spencer, N. M., & Pettitt, A. N. (2001) · Scandinavian Journal of Statistics · doi:10.1111/1467-9469.00225 · Cited by 17
Tran, M., Nott, D. J., & Leng, C. (2011) · Statistics and Computing · doi:10.1007/s11222-011-9279-3 · Cited by 15
Bayesian Variable Selection in Searching for Additive and Dominant Effects in Genome-Wide Data
Peltola, T., Marttinen, P., Jula, A. M., Salomaa, V. V., Perola, M., & Vehtari, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029115 · Cited by 15
Self-citationPeltola, T., Marttinen, P., & Vehtari, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049445 · Cited by 14
Self-citation
Cited by
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
Self-citationForecasting: theory and practice
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Ben Taieb, S., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., et al. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.11.001 · Cited by 921
Bayesian Item Response Modeling in R with brms and Stan
Bürkner, P.-C. (2021) · Journal of Statistical Software · doi:10.18637/jss.v100.i05 · Cited by 791
Using Stacking to Average Bayesian Predictive Distributions (with Discussion)
Yao, Y., Vehtari, A., Simpson, D., & Gelman, A. (2018) · Bayesian Analysis · doi:10.1214/17-ba1091 · Cited by 493
Self-citationCross validation for model selection: A review with examples from ecology
Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W. (2023) · Ecological Monographs · doi:10.1002/ecm.1557 · Cited by 459
Metadata
- Authors
- Piironen, J., & Vehtari, A.
- Venue
- Statistics and Computing
- Published
- 2016-04-07
- Volume
- 27
- Issue
- 3
- Pages
- 711-735
- ISSN
- 0960-3174 (ISSN-L)
- 1573-1375
- OpenAlex
- W1877880381
- Cited by
- 294
- References
- 51
- FWCI
- 23.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.