Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“Bernardo and Smith (1994) argue that suitable scoring rules for prediction are proper and local: propriety of the scoring rule motivates the decision maker to report his or her beliefs honestly, and for local scoring rules predictions are judged only on the plausibility they assign to the event…”

“The fit of model to data can be assessed using posterior predictive checks (Rubin, 1984), prior predictive checks (when evaluating potential replications involving new parameter values), or, more generally, mixed checks for hierarchical models (Gelman, Meng, and Stern, 2006).”

“In some ways, our paper is similar to the review article by Gelfand and Dey (1994), except that they were focused on model choice whereas our goal is more immediately to estimate predictive accuracy for the goal of model comparison.”

“…reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone, 1977).”

“We use the squared error as an example scoring function for point prediction, because the squared error and its derivatives seem to be the most common scoring functions in predictive literature (Gneiting, 2011).”

“Various difficulties have been noted with DIC (see Celeux et al., 2006, Plummer, 2008, and much of the discussion of Spiegelhalter et al., 2002) but there has been no consensus on an alternative.”

“Watanabe (2013) has also proposed a widely applicable Bayesian information criterion (WBIC) which works also in singular and unrealizable cases.”

“The logarithmic score is the unique (up to an affine transformation) local and proper scoring rule (Bernardo, 1979), and appears to be the most commonly used scoring rule in model selection.”

“For large n the difference is negligible, but for small n (or when using k-fold cross-validation) we can use a first order bias correction b by estimating how much better predictions would be obtained if conditioning on n data points (Burman, 1989):”

“Various difficulties have been noted with DIC (see Celeux et al., 2006, Plummer, 2008, and much of the discussion of Spiegelhalter et al., 2002) but there has been no consensus on an alternative.”

“There are extensions of AIC which have an adjustment related to the effective number of parameters (see Vehtari and Ojanen, 2012, section 5.5, and references therein) but these are seldom used due to stability problems and computational difficulties, issues that have motivated the construction of the…”

“From Rubin (1981).”

“Regular cross-validation and information criteria can be used for time series in case of stationary Markov process and squared error or a scoring function or rule which is well approximated by a quadratic form (Akaike, 1973, Burman et al., 1994).”

“Figure 2 is reminiscent of the direct likelihood methods of Dempster (1974) and Aitkin (2010).”

“A practical estimate of related sampling uncertainty can be obtained by analyzing the variation in the expected log predictive densities ̂ elppdi using parametric or nonparametric approaches (Vehtari and Lampinen, 2002).”

“One way to understand such differences is to calibrate based on simpler models (McCulloch, 1989).”

“DIC is a variation of the regularized information criteria which have been shown to be asymptotically equal to LOO-CV using plug-in predictive densities (Shibata 1989).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“So, in this case, DIC and AIC are the same.”

“Regular cross-validation and information criteria can be used for time series in case of stationary Markov process and squared error or a scoring function or rule which is well approximated by a quadratic form (Akaike, 1973, Burman et al., 1994).”

“Ando and Tsay (2010) have proposed an information criterion for the joint prediction, but its bias correction has the same computational difficulties as many other extensions of AIC and it can not be compared to cross-validation, since it is not possible to leave n data points out in the…”

“It is based on the ‘bread and peace’ model created by political scientist Douglas Hibbs (see Hibbs, 2008, for a recent review) to forecast elections based solely on economic growth (with corrections for wartime, notably Adlai Stevenson’s exceptionally poor performance in 1952 and Hubert Humphrey’s…”

“Hence the plan is to like to start with (5) and then apply some sort of bias correction to get a reasonable estimate of (2).”