Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“The selection of the most parsimonious model structure was based on three criteria: the Deviance Information Criteria (DIC) [24], the Watanabe-Akaike information criterion (WAIC) [25], and mean logarithmic scores [26].”

“The three models were compared using the Watan-abe-Akaike Information Criterion (WAIC) (Burnham and Anderson 2002; Gelman et al. 2014; Watanabe 2010).”

“The performances of the different candidate models were assessed by means of the deviance information criterion (DIC) (Spiegelhalter et al., 2002) and two varieties of the widely applicable information criterion (WAIC1 and WAIC2) defined in Gelman et al. (2014).”

“Bolded lines indicate best-fit model, as determined by lowest LOOIC score [41, 42].”

“The Deviance Information Criterion (DIC) [18] is DIC = ND + p D = 2p D + D ( N ϕ ) , and the Watanabe-Akaike information criterion (WAIC) [19] is (cid:14)(cid:8) (cid:15) (cid:8) (cid:16) (cid:8) (cid:17) For an interesting review of criteria for model selection, see also [20] and reference therein.”

“Compared to the Deviance Information Criteria (DIC), the WAIC has the advantage of being invariant to reparameterizations and is better suited for complex hierarchical models (Spiegelhalter et al., 2014; Gelman et al., 2014).”

“According to Gelman et al. (2014), the Watanabe-Akaike information criterion ( WAIC ) provides a fast and computationally convenient alternative to the cross-validation approach.”

“We used deviance information criteria (DIC, Spiegelhalter et al., 2002; Celeux et al., 2006) and Watanabe Akaike information criteria (WAIC, Watanabe and Opper, 2010; Gelman et al., 2014) to compare the two versions.”

“as a penalty term, where pWAIC2 is “the variance of individual terms in the log predictive density summed over the n data points” [48].”

“WAIC is based on the overall likelihood of the model and has a penalty term for the effective number of free parameters (Gelman et al., 2014).”

“By determining the mean log-likelihoods of the parameters obtained from the posterior distribution, the Watanabe–Akaike Information Criterion (WAIC) can be obtained (19).”

“The Bayes estimator has been shown to concentrate about the parameter value which minimizes KL-divergence between a model and the true distribution as sample size increases (Gelman et al., 2014; Zellner, 1988).”

“…to many interpretations and controversies, as illustrated by the recent American Statistical Association statement on the dangers of over-interpreting p-values (Wasserstein and Lazar, 2016) and other calls (Benjamin et al., 2018; Gelman and Robert, 2014; Johnson, 2013; McShane et al., 2019).”

“WAIC [29] is also another popular fully Bayes information criteria.”

“𝑊𝑊𝑊𝑊𝐼𝐼𝐶𝐶 𝑊𝑊𝑊𝑊𝐼𝐼𝐶𝐶 is given by the computed log pointwise posterior predictive density ( ) plus an effective number of parameters ( ) to penalise overfitting (Gelman, Hwang, & Vehtari, 2014): 𝑊𝑊𝑊𝑊𝐼𝐼𝐶𝐶 where the computed .”

“We suggest to determine c2 using the Bayesian cross-validation method (Gelman et al., 2014), with the other hyperparameters γ = 0.1 and c1 = 1 fixed.”

“The resulting WAIC model weight was used as a neuron-level model selection criterion, and only neurons for which the full model was favoured over baseline were retained for subsequent analyses.”

“Watanabe-Akaike Information Criterion / Widely Applicable Information Criterion (WAIC) (Watanabe, 2010) is calculated as with lppd the log posterior predictive density (lppd) and the effective number of parameters p WAIC defined using equation (11) of Gelman et al. (2014).”

“1; Watanabe, 2010; Gelman et al., 2014).”

“Moreover, many IC are plug-in scores evaluated at point estimates, discarding posterior uncertainty [45], and even defining the effective sample size for penalization is non-trivial given temporally correlated TACs and implementation details such as up-sampled convolution or basis-function…”

“60 , 61 WAIC is a fully Bayesian method and preferable to deviance information criterion (DIC) in Bayesian modeling, 62 especially with MCMC sampling, as it uses the entire posterior distribution and is asymptotically equivalent to Bayesian cross-validation, while DIC is not considered a singular…”

“In particular, we have employed the specific version of WAIC (pWAIC2) proposed by Gelman et al. [ 29], which is recommended for being closer in practice to a leave- one-out cross-validation analysis.”

“Model comparisons were based on Watanabe-Akaike Information Criterion (WAIC), an estimate of out-of-sample prediction accuracy for Bayesian models [48].”

“This is particularly relevant for models with hierarchical priors, where the extent of hierarchical shrinkage (i.e., the influence of the priors) depends on the data itself (Gelman et al, 2014).”

“This criterion balances model fit and complexity, if a variable is not relevant, the WAIC value increases significantly [66, 67].”

“The optimal model was identified based on the Watanabe–Akaike Information Criterion, which provided the best balance of fit and parsimony [22 ].”

“It is also asymptotically equivalent to leave-one-out cross-validation (LOO-CV) under regular conditions, making it a widely accepted tool for Bayesian model comparison [25,26].”

“In particular, we have employed the specific version of WAIC (pWAIC2) proposed by Gelman et al. (2014), which is recommended for being closer in practice to a leave-one-out cross-validation analysis.”

“We found that the GP model consistently outperformed the MLR model when we compared the two in terms of the “widely applicable” information criterion (WAIC; Watanabe, 2013; Gelman, Hwang, & Vehtari, 2014).”

“We use the Expected Log Probability Distribution (ELPD; see Vehtari et al. 2016; Gelman et al. 2013) to determine which kinematic model is preferred: (5) where P ( x i | x − i ) is the probability of observing the i - th data point given the model that excludes this data point and N is the number…”

“The correlation term is easily interpretable and is a common goodness-of-fit statistic used in many settings, including Bayesian spatiotemporal modelling [38].”

“We use WAIC mainly because, according to Gelman et al. 79 , it is a fast and computationally convenient alternative, is fully Bayesian (using the posterior distribution rather than a point estimate), gives reasonable results, and has a more-or-less explicit connection to cross-validation.”

“Whereas less informative priors tend to produce posterior distributions that are more influenced by the data and shaped by the likelihood function (Depaoli & van de Schoot, 2017; Gelman et al., 2013).”

“A detailed methodological discussion of these criteria is provided in Gelman et al. (2014b).”

“To account for this, model complexity metrics such as the Bayesian information criterion (Neath and Cavanaugh, 2012) or external validation metrics such as cross-validation information criterion (Gelman et al., 2014), could be incorporated into future model evaluations.”

“However, DIC has its own limitations (Gelman et al., 2014).”

“Similarly, for WAIC, a difference of 10 is considered substantial evidence in favor of the model with the lower WAIC [60].”

“Models that best fi t the data were selected based on Widely Applicable Information Criteria (WAIC; Gelman et al., 2014) measures.”

“…(Held, Schr¨odle, & Rue, 2010; Pettit, 1990), and the Watanabe-Akaike Information criterion (WAIC, Watan-abe, 2013), following the formulation of Gelman, Hwang, and Vehtari (2014), the Deviance Information Criterion (DIC, Spiegelhalter, Best, Carlin, & Van Der Linde, 2002), alongside with the…”

“Specifically, we perform stepwise forward selection based on the Watanabe-Akaike Information Criterion (WAIC) (Watanabe, 2013; Gelman et al., 2014).”

“…in a model passed the ˆ R threshold, we evaluated model fit and predictive performance using leave-one-out cross-validation (LOO-CV) with Pareto smoothed importance sampling to estimate the expected log predictive density (ELPD), using the loo package (Gelman et al. 2014, Vehtari et al. 2017).”

“In comparison to the ORL model, the data sets in Group C demonstrate a superior fit for the VSE model, as evidenced by the AIC, BIC, and WAIC measures.”

“The elpd loo describes a model ’ s accuracy in predicting future, out-of-sample data and thus captures the goodness of fit of the model while also controlling for overfitting (Gelman et al., 2014).”

“The two most popular criteria for this purpose are the Bayesian leave-one-out information criterion (LOOIC) and the widely applicable information criterion (WAIC) (Watanabe 2010; Gelman et al. 2014; Vehtari et al. 2017).”

“Moreover, an SC-DPM prior has to be placed on the subject’s true score { θ i } I 1 to fix their mean and resolve identifiability issues (Gelman et al., 2014), as a by-product under the parameter-expanded specification, equation (43) becomes for i = 1 ,..., I ; j ∈ R i .”

“…of the covariance matrices and, thus, the SEA B were sampled using three Markov Chain Monte Carlo algorithms that were run until convergence, i.e., Gelman-Rubin statistics < 1.01 and effective sample sizes greater than 2000 for every parameter (Gelman et al. 2014; van de Schoot et al. 2021).”

“Table 3 presents the difference in Watanabe-Akaike Information Criterion scores (WAIC, Watanabe and Opper 2010; Watanabe 2013; Gelman et al. 2014), as well as the posterior mean and 90% credible intervals for the Spearman correlation between the logistic unfolding model (LUM) and both the probit…”

“A lower WAIC value indicates a better model fit in terms of parsimony and superior predictive capacity compared to a model with a higher WAIC value (Gelman et al. 2013; Vehtari et al. 2017).”

“The leave-one-out cross validation (LOOCV) [24, 63, 27] is an alternative method to estimate the generalization error. where is the sample leaving X i out.”

“Lastly, hyperparameters corresponding to the same unique realization of spline coefficients would be identical, that is”

“This ensures that the desired acceptance rate stays around 0.44; see [14] for pertinent details.”

“| Since posterior is intractable under these priors, one can construct a blocked Gibbs sampling algorithm and execute it using MCMC simulation (Gelman et al. 2014) which iterates between the following pair of conditional distributions, Instead of MCMC simulations, the INLA approach provides…”

“We use seven latent classes for the global clustering stage as determined by the LOO-CV metric.”

“The LOO-CV rests on the derivation of the expected log point-wise predictive density (ELPD) for new data defined as where p t ð ~ y i Þ represents the distribution of the true but unknown data-generating process for the predicted values ~ y i and where Equation (13) is approximated by cross-validation procedures.”

“The Deviance Information Criterion (DIC) is used to compare the two models (Gelman et al., 2014).”

“However, it can still encounter difficulties in specific scenarios as it depends on a data partition that may pose challenges for structured models, such as those involving spatial or network data (Gelman et al., 2013).”

“In our Bayesian framework, we can use the Watanabe-Akaike information criterion (WAIC) [28] to compare different model specifications.”

“WAIC takes into account both quality of fit and number of parameters and has the following expression: where N VEGF is the number of VEGF data points for each cell type, [35].”