Multimodel Inference: Understanding AIC and BIC in Model Selection
The DOI matches its Crossref record, with a small difference in title or year; 59 of the 59 references it cites were found in a registry or an index.
Abstract
The model selection literature has been generally poor at reflecting the deep foundations of the Akaike information criterion (AIC) and at making appropriate comparisons to the Bayesian information criterion (BIC). There is a clear philosophy, a sound criterion based in information theory, and a rigorous statistical foundation for AIC. AIC can be justified as Bayesian using a “savvy” prior on models that is a function of sample size and the number of model parameters. Furthermore, BIC can be derived as a non-Bayesian result. Therefore, arguments about using AIC versus BIC for model selection cannot be from a Bayes versus frequentist perspective. The philosophical context of what is assumed about reality, approximating models, and the intent of model-based inference should determine whether AIC or BIC is used. Various facets of such multimodel inference are presented here, particularly methods of model averaging.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The weight of evidence was calculated, which denotes the probability that a model is the most plausible of the set.”
From Academic Achievement Trajectories of Homeless and Highly Mobile Students: Resilience in the Context of Chronic and Acute Risk (2013) · cites this paper as Burnham & Anderson, 2004
In the citing paper
“The average AIC is 0.5 less than the average AIC for the classic Ebbinghaus functions, which—though it indicates a better fit—may not be considered a meaningful difference; a difference of 2 is considered 'meaningful' [49].”
From Replication and Analysis of Ebbinghaus’ Forgetting Curve (2015) · cites this paper as [49]
In the citing paper
“Because AIC effectively offers a relative estimate of the information lost, the model with the lowest score gives the best representation of the data and is thus considered most likely.”
From Rapid intra-adrenal feedback regulation of glucocorticoid synthesis (2015) · cites this paper as [28]
Citations
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · cited 1×
“UNDERSTANDING BIC Schwarz (1978) derived the Bayesian information criterion as BIC 2ln( ) log( ) .œ _ K n As usually used one computes BIC for each model and selects the model with the smallest criterion value.”
MethodModel selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495
Self-citationGelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 1×
“The MLE ( ) converges, as gets large, to the that minimizes K-L information loss for model)̂ )y n o g. Large sample expected AIC converges to E(AIC) 2 2 ( , ( )) .œ † ± C I f g K )o”
BackgroundBayesian model selection in social research
Raftery, A. E. (1995) · Sociological Methodology · doi:10.2307/271063 · Cited by 6,823 · cited 2×
“=1 (Raftery 1995).”
BackgroundRegression and time series model selection in small samples
HURVICH, C. M., & TSAI, C.-L. (1989) · Biometrika · doi:10.1093/biomet/76.2.297 · Cited by 6,586
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 1×
“…that has evaluated, or even considered, model averaged inferences compared to the best model strategy that model averaging is superior (e.g., Buckland et al. 1997; Hoeting et al. 1999; Wasserman 2000; Breiman 2001; Burnham and Anderson 2002; Hansen and Kooperberg 2002; Lukacs et al. in review).”
ResultHoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118 · cited 4×
“The literature (e.g., Hoeting et al. 1999) supports that the measurements and ( , , ) on a subject cany x x x œ 1 13á w be suitably modeled as multivariate normal.”
BackgroundMethodResultInfluentialBurnham, K. P., & Anderson, D. R. (1998) · doi:10.1007/978-1-4757-2917-7 · Cited by 3,387
Self-citationInformation Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_38 · Cited by 3,028
Information Theory and an Extention of the Maximum Likelihood Principle
(1973) · IEEE International Symposium on Information Theory (ISIT) · Cited by 2,794
Information theory as an extension of the maximum likelihood principle
Akaike, H. (1973) · Medical Entomology and Zoology · Cited by 2,357
Further analysis of the data by Akaike's information criterion and the finite corrections
Sugiura, N. (1978) · Communications in Statistics - Theory and Methods · doi:10.1080/03610927808827599 · Cited by 2,144 · cited 1×
“…includes when is small, for any ) there is a small sample (second order biasn K correction) version called AIC ,- AIC 2log( ( )) 2^- œ _ ) K 2 ( 1)1 K K n K (see Sugiura 1978; Hurvich and Tsai 1989, 1995), and this should be used unless n/K K about 40 for the model with the largest value of .”
MethodModel Selection: An Integral Part of Inference
Buckland, S. T., Burnham, K. P., & Augustin, N. H. (1997) · Biometrics · doi:10.2307/2533961 · Cited by 1,717 · cited 5×
“, Buckland et al. 1997; Hoeting et al. 1999; Wasserman 2000; Breiman 2001; Burnham and Anderson 2002; Hansen and Kooperberg 2002). The method known as boosting is a type of model averaging (Hand and Vinciotti 2003:130; this article is also useful reading for its comments on truth and models). However, model-averaged i…”
MethodResultInfluentialSelf-citationAvoiding Pitfalls When Using Information-Theoretic Methods
Anderson, D. R., & Burnham, K. P. (2002) · The Journal of Wildlife Management · doi:10.2307/3803155 · Cited by 1,324
Self-citationStatistical modeling: The two cultures
Breiman, L. (2001) · Cited by 1,314
An asymptotic equivalence of choice of model by cross validation and Akaike's criterion
Stone, M. (1977) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1977.tb01603.x · Cited by 1,286
Likelihood of a model and information criteria
Akaike, H. (1981) · Journal of Econometrics · doi:10.1016/0304-4076(81)90071-3 · Cited by 1,274
Bayesian Model Choice: Asymptotics and Exact Calculations
Gelfand, A. E., & Dey, D. K. (1994) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1994.tb01996.x · Cited by 1,209
Chamberlin, T. C. (1965) · Science · doi:10.1126/science.148.3671.754 · Cited by 1,069
Bayesian Model Selection and Model Averaging
Wasserman, L. (2000) · Journal of Mathematical Psychology · doi:10.1006/jmps.1999.1278 · Cited by 986 · cited 1×
“…that has evaluated, or even considered, model averaged inferences compared to the best model strategy that model averaging is superior (e.g., Buckland et al. 1997; Hoeting et al. 1999; Wasserman 2000; Breiman 2001; Burnham and Anderson 2002; Hansen and Kooperberg 2002; Lukacs et al. in review).”
ResultLinhart, H., & Zucchini, W. (1986) · Cited by 869
Statistical Evidence: A Likelihood Paradigm
Reeves, J. H., & Royall, R. M. (1998) · Journal of the American Statistical Association · doi:10.2307/2669869 · Cited by 814
Selected Papers of Hirotugu Akaike
(1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0 · Cited by 791
The Journal of Wildlife Management
(2011) · Journal of Wildlife Management · doi:10.1002/(issn)1937-2817 · Cited by 705
Cross-Validatory Choice and Assessment of Statistical Predictions (With Discussion)
Stone, M. (1976) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1976.tb01573.x · Cited by 686
FORSTER, M., & SOBER, E. (1994) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/45.1.1 · Cited by 659
Approximate Bayes factors and accounting for model uncertainty in generalised linear models
Raftery, A. (1996) · Biometrika · doi:10.1093/biomet/83.2.251 · Cited by 633 · cited 1×
“3 counterparts in the Bayesian literature (Raftery 1996).”
BackgroundStatistical Evidence: A Likelihood Paradigm
Vieland, V. J., & Hodge, S. E. (1998) · The American Journal of Human Genetics · doi:10.1086/499065 · Cited by 606
Regression and Time Series Model Selection
McQuarrie, A. D. R., & Tsai, C.-L. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/3573 · Cited by 513 · cited 1×
“This is true even for major or recent publications, e.g., Linhart and Zucchini (1986), McQuarrie and Tsai (1998), and Lahiri (2001).”
BackgroundKey Concepts in Model Selection: Performance and Generalizability
Forster, M. R. (2000) · Journal of Mathematical Psychology · doi:10.1006/jmps.1999.1284 · Cited by 384 · cited 3×
“Parsimony enjoys a featured place in scientific thinking in general and in modeling specifically (see Forster and Sober 1994; Forster 2000, 2001) for a strictly science philosophy perspective).”
BackgroundAkaike, H. (1985) · A Celebration of Statistics · doi:10.1007/978-1-4613-8560-8_1 · Cited by 360 · cited 1×
“Akaike (1973, 1974, 1985, 1994) showed that the critical issue for getting a rigorous model selection criterion based on K-L information was to estimate E E [log( ( ( )))]^y x g x y± ) where the inner part is just E [log( ( ))] with replaced by the maximum likelihood0 g x ± ) ) estimator of based…”
BackgroundEdwards, A. W. F. (1992) · Johns Hopkins University Press eBooks · doi:10.56021/9780801844454 · Cited by 358
A Critique of the Bayesian Information Criterion for Model Selection
WEAKLIEM, D. L. (1999) · Sociological Methods & Research · doi:10.1177/0049124199027003002 · Cited by 349
Journal of Statistics Education
(2007) · Encyclopedia of Measurement and Statistics · doi:10.4135/9781412952644.n236 · Cited by 348
Model Selection for Extended Quasi-Likelihood Models in Small Samples
Hurvich, C. M., & Tsai, C.-L. (1995) · Biometrics · doi:10.2307/2533006 · Cited by 305
Breiman, L. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10475276 · Cited by 282 · cited 2×
“These variance^ -̂) )3 # components are multiplied by the Akaike weights, which reflect the relative support, or evidence, for model .”
BackgroundGeorge, E. I. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess0989.pub2 · Cited by 188
deLeeuw, J. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_37 · Cited by 184 · cited 2×
“- Akaike (1973, 1974) found a formal relationship between K-L information (a dominant paradigm in information and coding theory) and likelihood theory (the dominant paradigm in statistics) (see deLeeuw 1992).”
BackgroundWeighing the Odds: A Course in Probability and Statistics
Williams, D. S. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9781139164795 · Cited by 178 · cited 1×
“From a recent book (Williams 2001, p. 235), and quoted exactly, “ á á we CANNOT ignore the degree of resolution of the experiment when choosing our prior."”
BackgroundGeneralizing the derivation of the schwarz information criterion
Cavanaugh, J. E., & Neath, A. A. (1999) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929908832282 · Cited by 169
Fitting Percentage of Body Fat to Simple Body Measurements
Johnson, R. W. (1996) · Journal of Statistics Education · doi:10.1080/10691898.1996.11910505 · Cited by 155
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151 · cited 2×
“In particular, we note the recent development of the deviance information criterion (DIC) by Spiegelhalter et al. (2002). As these authors note, DIC behaves like AIC, not like BIC which is one reason they prefer DIC (it avoids the defects of BIC model selection).”
BackgroundStatistical Inference Based on the likelihood
(2017) · Routledge eBooks · doi:10.1201/9780203738627 · Cited by 143
Model Specification: The Views of Fisher and Neyman, and Later Developments
Lehmann, E. L. (1990) · Statistical Science · doi:10.1214/ss/1177012164 · Cited by 136
Statistical Inference Based on the Likelihood.
Iyengar, S., & Azzalini, A. (1997) · Journal of the American Statistical Association · doi:10.2307/2965597 · Cited by 114
Local Versus Global Models for Classification Problems: Fitting Models Where it Matters
Hand, D. J., & Vinciotti, V. (2003) · The American Statistician · doi:10.1198/0003130031423 · Cited by 101
Spline Adaptation in Extended Linear Models (with comments and a rejoinder by the authors
Hansen, M. H., & Kooperberg, C. L. (2002) · Statistical Science · doi:10.1214/ss/1023798997 · Cited by 90 · cited 1×
“…that has evaluated, or even considered, model averaged inferences compared to the best model strategy that model averaging is superior (e.g., Buckland et al. 1997; Hoeting et al. 1999; Wasserman 2000; Breiman 2001; Burnham and Anderson 2002; Hansen and Kooperberg 2002; Lukacs et al. in review).”
ResultSimplicity, Inference and Modelling: Keeping It Sophisticatedly Simple
Nikolopoulos, K. I. (2003) · International Journal of Forecasting · doi:10.1016/s0169-2070(03)00018-9 · Cited by 82
Davison, A. C., & Firth, D. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.02022 · Cited by 77
Predictive Variable Selection in Generalized Linear Models
Meyer, M. C., & Laud, P. W. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502388618654 · Cited by 41
INFORMATION THEORY AS AN EXTENSION OF THE MAXIMUM LIKELIHOOD
Akaike, H. (1973) · Cited by 27
(2001) · SPIE eBooks · doi:10.1214/lnms/1215540957 · Cited by 24
Prediction with vague prior knowledge
Reschenhofer, E. (1996) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929608831716 · Cited by 20 · cited 1×
“Their target models are different (Reschenhofer 1996).”
International Statistical Institute
(1887) · Science · doi:10.1126/science.ns-9.225.507-a · Cited by 7
Research works printed but not linked: 7
- Akaike, Hirotugu, 1973, Second International Symposium on Information Theory, 267
- Akaike, Hirotugu, 1994, Engineering and Scientific Applications: Vol. 3. Proceedings of the First US/Japan Conference on the Frontiers of Statistical Modeling: An Informational Approach, 27
- Boltzmann, Ludwig, 1877, Wiener Berichte, 76, 373
- Breiman, Leo, 2001, Statistical Science, 26, 199
- Hoeting, Jennifer A., 1999, ” Statistical Science, 14, 382
- Key, Jane T., 1999, Bayesian Statistics 6, 343
- Takeuchi, Kei, 1976, Suri-Kagaku (Mathematic Sciences), 153, 12
Cited by
Partial Least Squares Structural Equation Modeling (PLS-SEM) Using R: A Workbook
Hair, J. F., Hult, G. T. M., Ringle, C. M., Sarstedt, M., Danks, N. P., & Ray, S. (2021) · Classroom Companion: Business · doi:10.1007/978-3-030-80519-7 · Cited by 10,343
Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1029-6 · Cited by 4,104
Self-citationThe effect of human mobility and control measures on the COVID-19 epidemic in China
Kraemer, M. U. G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M., Open COVID-19 Data Working Group, du Plessis, L., Faria, N. R., Li, R., et al. (2020) · Science · doi:10.1126/science.abb4218 · Cited by 2,950
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583
“For a detailed discussion of approaches that account for the complexity of parametric models, see Burnham and Anderson (2002) or Burnham and Anderson (2004).”
Symonds, M. R. E., & Moussalli, A. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1037-6 · Cited by 2,543
Metadata
- Authors
- Burnham, K. P., & Anderson, D. R.
- Affiliations
- United States Geological Survey (US)
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2004-11
- Volume
- 33
- Issue
- 2
- Pages
- 261-304
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2158196600
- Semantic Scholar
- 9bea108c81919453a5774bfdc3586a4e03e52338
- Cited by
- 11,648
- References
- 58
- Citations by year
- 2026: 825 · 2025: 929 · 2024: 944 · 2023: 884 · 2022: 906 · 2021: 1070 · 2020: 935 · 2019: 803 · 2018: 662 · 2017: 606 · 2016: 567 · 2015: 490 · 2014: 474 · 2013: 416 · 2012: 283
- FWCI
- 49.63
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Sociological Methods & Research (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- multi-model inference, model selection, Akaike information criterion, Bayesian information criterion, model averaging, information theory, prior distributions, model-based inference, philosophy of statistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.