The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.
Model Selection Techniques: An Overview
Open accessAbstract
In the era of big data, analysts usually explore various statistical models or machine-learning methods for observed data to facilitate scientific discoveries or gain predictive power. Whatever data and fitting procedures are employed, a crucial step is to select the most appropriate model or method from a set of candidates. Model selection is a key ingredient in data analysis for reliable and reproducible statistical inference or prediction, and thus it is central to scientific studies in such fields as ecology, economics, engineering, finance, political science, biology, and epidemiology. There has been a long history of model selection techniques that arise from researches in statistics, information theory, and signal processing. A considerable number of methods has been proposed, following different philosophies and exhibiting varying performances. The purpose of this article is to provide a comprehensive overview of them, in terms of their motivation, large sample performance, and applicability. We provide integrated and practically relevant discussions on theoretical properties of state-of-the-art model selection approaches. We also share our thoughts on some controversial views on the practice of model selection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
Fan, J., & Li, R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382273 · Cited by 8,900
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 7,715
The adaptive lasso and its oracle properties
Zou, H. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000735 · Cited by 7,470
Model Selection and Estimation in Regression with Grouped Variables
Yuan, M., & Lin, Y. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00532.x · Cited by 7,261
Decoding by Linear Programming
Candes, E. J., & Tao, T. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.858979 · Cited by 7,121
Regression and time series model selection in small samples
HURVICH, C. M., & TSAI, C.-L. (1989) · Biometrika · doi:10.1093/biomet/76.2.297 · Cited by 6,586
An Introduction to Variable and Feature Selection.
Guyon, I., & Elisseeff, A. (2003) · Cited by 6,576
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.
Daubechies, I., Defrise, M., & De Mol, C. (2004) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20042 · Cited by 4,958
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“It has been reported that the selected variables from these penalized regression methods are often severely unstable, in the sense that the selection results can be drastically different under a tiny perturbation of data (see [78] and the references therein).”
BackgroundPati, Y. C., Rezaiifar, R., & Krishnaprasad, P. S. (2002) · Proceedings of 27th Asilomar Conference on Signals, Systems and Computers · doi:10.1109/acssc.1993.342465 · Cited by 4,323
Vaart, A. W. V. D. (1998) · doi:10.1017/cbo9780511802256 · Cited by 4,159
Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
Golub, G. H., Heath, M., & Wahba, G. (1979) · Technometrics · doi:10.1080/00401706.1979.10489751 · Cited by 3,934
Model selection in ecology and evolution
Johnson, J. B., & Omland, K. S. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2003.10.013 · Cited by 3,838
Nearly unbiased variable selection under minimax concave penalty
Zhang, C.-H. (2010) · The Annals of Statistics · doi:10.1214/09-aos729 · Cited by 3,835
The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses
Wilks, S. S. (1938) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177732360 · Cited by 3,493
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (2015) · Measures of Complexity · doi:10.1007/978-3-319-21852-6_3 · Cited by 3,105
The Determination of the Order of an Autoregression
Hannan, E. J., & Quinn, B. G. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01072.x · Cited by 2,992
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · cited 1×
“Cross-validation (CV) [38], [39] is a class of model selection methods widely used in machine learning practice.”
MethodStatistical Learning with Sparsity: The Lasso and Generalizations
Hastie, T., Tibshirani, R., & Wainwright, M. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b18401 · Cited by 2,520
Akaike's Information Criterion in Generalized Estimating Equations
Pan, W. (2001) · Biometrics · doi:10.1111/j.0006-341x.2001.00120.x · Cited by 2,493
Fitting autoregressive models for prediction
Akaike, H. (1969) · Annals of the Institute of Statistical Mathematics · doi:10.1007/bf02532251 · Cited by 2,471
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (1971) · Theory of Probability & Its Applications · doi:10.1137/1116025 · Cited by 2,448
Simultaneous analysis of Lasso and Dantzig selector
Bickel, P. J., Ritov, Y., & Tsybakov, A. B. (2009) · The Annals of Statistics · doi:10.1214/08-aos620 · Cited by 2,396
Meinshausen, N., & Bühlmann, P. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00740.x · Cited by 2,215
Modeling and variable selection in epidemiologic analysis
Greenland, S. (1989) · American Journal of Public Health · doi:10.2105/ajph.79.3.340 · Cited by 2,208
Unsupervised learning of finite mixture models
Figueiredo, M. A. T., & Jain, A. K. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.990138 · Cited by 2,125
Mallows, C. L. (1995) · Technometrics · doi:10.1080/00401706.1995.10484370 · Cited by 2,083
Fitting Autoregreesive Models for Prediction
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_10 · Cited by 2,019
Extended Bayesian information criteria for model selection with large model spaces
Chen, J., & Chen, Z. (2008) · Biometrika · doi:10.1093/biomet/asn034 · Cited by 2,011
Mallows, C. L. (1973) · Technometrics · doi:10.2307/1267380 · Cited by 1,975
Uncertainty principles and ideal atomic decomposition
Donoho, D. L., & Huo, X. (2001) · IEEE Transactions on Information Theory · doi:10.1109/18.959265 · Cited by 1,949
The Dantzig selector: Statistical estimation when p is much larger than n
Candes, E., & Tao, T. (2007) · The Annals of Statistics · doi:10.1214/009053606000001523 · Cited by 1,747
Orthogonal least squares methods and their application to non-linear system identification
Chen, S., Billings, S. A., & Luo, W. (1989) · International Journal of Control · doi:10.1080/00207178908953472 · Cited by 1,540
The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
Allen, D. M. (1974) · Technometrics · doi:10.1080/00401706.1974.10489157 · Cited by 1,469 · cited 1×
“Cross-validation (CV) [38], [39] is a class of model selection methods widely used in machine learning practice.”
MethodStoica, P., & Selen, Y. (2004) · IEEE Signal Processing Magazine · doi:10.1109/msp.2004.1311138 · Cited by 1,358
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286
Dawid, A. P. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981683 · Cited by 1,085
The minimum description length principle in coding and modeling
Barron, A., Rissanen, J., & Yu, B. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.720554 · Cited by 1,019
Minimax estimation via wavelet shrinkage
Donoho, D. L., & Johnstone, I. M. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691081 · Cited by 1,005
On Model Selection Consistency of Lasso
ZhaoPeng, & Yubin, Y. (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248637 · Cited by 996
An Information Measure for Classification
Wallace, C. S., & Boulton, D. M. (1968) · The Computer Journal · doi:10.1093/comjnl/11.2.185 · Cited by 933
Stochastic Complexity and Modeling
Rissanen, J. (1986) · The Annals of Statistics · doi:10.1214/aos/1176350051 · Cited by 922
Selection of the order of an autoregressive model by Akaike's information criterion
SHIBATA, R. (1976) · Biometrika · doi:10.1093/biomet/63.1.117 · Cited by 789
Model Selection and the Principle of Minimum Description Length
Hansen, M. H., & Yu, B. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753168398 · Cited by 788
AN ASYMPTOTIC THEORY FOR LINEAR MODEL SELECTION
Shao, J. (1997) · Cited by 737
Statistical Learning with Sparsity: The Lasso and Generalizations
O'Brien, C. M. (2016) · International Statistical Review · doi:10.1111/insr.12167 · Cited by 726
Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0<q⩽1
Foucart, S., & Lai, M.-J. (2009) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2008.09.001 · Cited by 715
Variable selection in regression—a tutorial
Andersen, C. M., & Bro, R. (2010) · Journal of Chemometrics · doi:10.1002/cem.1360 · Cited by 697
Risk bounds for model selection via penalization
Barron, A., Birgé, L., & Massart, P. (1999) · Probability Theory and Related Fields · doi:10.1007/s004400050210 · Cited by 657
Yang, Y. (2005) · Biometrika · doi:10.1093/biomet/92.4.937 · Cited by 605
Self-citationValid post-selection inference
Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013) · The Annals of Statistics · doi:10.1214/12-aos1077 · Cited by 599
An optimal selection of regression variables
SHIBATA, R. (1981) · Biometrika · doi:10.1093/biomet/68.1.45 · Cited by 562
Shibata, R. (1980) · The Annals of Statistics · doi:10.1214/aos/1176344897 · Cited by 542
The Risk Inflation Criterion for Multiple Regression
Foster, D. P., & George, E. I. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325766 · Cited by 541
Methods and Criteria for Model Selection
Kadane, J. B., & Lazar, N. A. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000269 · Cited by 429
Cross-validation for selecting a model selection procedure
Zhang, Y., & Yang, Y. (2015) · Journal of Econometrics · doi:10.1016/j.jeconom.2015.02.006 · Cited by 373
Self-citationAccurate estimation of influenza epidemics using Google search data via ARGO
Yang, S., Santillana, M., & Kou, S. C. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1515373112 · Cited by 373
Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression
Nishii, R. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346522 · Cited by 335
Yang, Y. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753168262 · Cited by 312
Self-citationSparse estimators and the oracle property, or the return of Hodges’ estimator
Leeb, H., & Pötscher, B. M. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.017 · Cited by 229
On Predictive Least Squares Principles
Wei, C. Z. (1992) · The Annals of Statistics · doi:10.1214/aos/1176348511 · Cited by 194
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151
Asymptotic MAP criteria for model selection
Djurić, P. M. (1998) · IEEE Transactions on Signal Processing · doi:10.1109/78.720374 · Cited by 149
Ing, C., & Lai, T. L. (2011) · Statistica Sinica · doi:10.5705/ss.2010.081 · Cited by 137
Finite sample criteria for autoregressive order selection
BROERSEN, P. M. T. (2000) · IEEE Transactions on Signal Processing · doi:10.1109/78.887047 · Cited by 131
Estimation Stability With Cross-Validation (ESCV)
Lim, C., & Yu, B. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2015.1020159 · Cited by 126
Rodeo: Sparse, greedy nonparametric regression
Lafferty, J., & Wasserman, L. (2008) · The Annals of Statistics · doi:10.1214/009053607000000811 · Cited by 125
van Erven, T., Grünwald, P., & de Rooij, S. (2012) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2011.01025.x · Cited by 111
Bridging AIC and BIC: A New Criterion for Autoregression
Ding, J., Tarokh, V., & Yang, Y. (2017) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2717599 · Cited by 106
Self-citationRodeo: Sparse, greedy nonparametric regression
Lafferty, J., & Wasserman, L. (2018) · KiltHub Repository · doi:10.1184/r1/6609134.v1 · Cited by 105
Model selection by MCMC computation
Andrieu, C., Djurić, P. M., & Arnaud Doucet, B. (2001) · Signal Processing · doi:10.1016/s0165-1684(00)00188-2 · Cited by 94
Parry, M., Dawid, A. P., & Lauritzen, S. L. (2012) · The Annals of Statistics · doi:10.1214/12-aos971 · Cited by 92
Parametric or nonparametric? A parametricness index for model selection
Liu, W., & Yang, Y. (2011) · The Annals of Statistics · doi:10.1214/11-aos899 · Cited by 65
Self-citationPREDICTION/ESTIMATION WITH SIMPLE LINEAR MODELS: IS IT REALLY THAT SIMPLE?
Yang, Y. (2006) · Econometric Theory · doi:10.1017/s0266466607070016 · Cited by 57
Self-citationConfidence sets for model selection by F -testing
Ferrari, D., & Yang, Y. (2014) · Statistica Sinica · doi:10.5705/ss.2014.110 · Cited by 57
Self-citationVariable Selection Diagnostics Measures for High-Dimensional Regression
Nan, Y., & Yang, Y. (2013) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2013.829780 · Cited by 53
Self-citationComparing Learning Methods for Classification
Yang, Y. (2006) · Cited by 45
Self-citationPost-selection adaptive inference for Least Angle Regression and the LassoPreprint
Taylor, J., Lockhart, R., Tibshirani, R. J., & Tibshirani, R. J. (2014) · arXiv (Cornell University) · Cited by 35
Bayesian Model Comparison with the Hyvärinen Score: Computation and Consistency
Shao, S., Jacob, P. E., Ding, J., & Tarokh, V. (2018) · Figshare · doi:10.6084/m9.figshare.7063805 · Cited by 28
Self-citationBayesian Model Comparison with the Hyvärinen Score: Computation and Consistency
Shao, S., Jacob, P. E., Ding, J., & Tarokh, V. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2018.1518237 · Cited by 26
Self-citationToward an Objective and Reproducible Model Choice via Variable Selection Deviation
Yang, W., & Yang, Y. (2016) · Biometrics · doi:10.1111/biom.12554 · Cited by 15
Self-citationSLANTS: Sequential Adaptive Nonlinear Modeling of Time Series
Han, Q., Ding, J., Airoldi, E. M., et al. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2716898 · Cited by 14
Self-citation
Metadata
- Authors
- Ding, J., Tarokh, V., & Yang, Y.
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-11
- Volume
- 35
- Issue
- 6
- Pages
- 16-34
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2897173776
- Cited by
- 383
- References
- 91
- FWCI
- 20.36
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.