Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“This article, however, considers a transformed version of Watanabe and Opper (2010)’s WAIC that was initially proposed by Gelman et al. (2014a) and Gelman et al. (2014b).”

“To evaluate the expectation in the ELPPD on the target data we use a Monte Carlo approximation, which results in the computed log pointwise predictive density (CLPPD, Gelman et al., 2014) estimated by,”

“WAIC, widely known as a robust Bayesian information criterion [47], is rooted in the harmonious blend of pointwise predictive density and model complexity.”

“WAIC is a more generalized version of the AIC (Watanabe and Opper 2010; Gelman et al., 2014b), and both are estimates of the predictive performance of the model under leave-one-out cross-validation.”

“This greatly improves the accuracy of quantification while also reducing the number of effective parameters (Gelman et al., 2014; Vehtari et al., 2017), without introducing additional PK assumptions.”

“By multiplying 𝜙 at both sides of the second display of (7), it is evident that 𝜙 𝜷 carries population-averaged interpretation since 𝜙𝑢 ( 𝑠 𝑖 ) + 𝜙𝜖 𝑖𝑗 follows the standard logistic distribution.”

“As outlined in Gelman et al. (2014), the WAIC on the deviance scale can be estimated using the following formula: Here, 𝜃 ( s ) } Ss = 1 represents S draws from the posterior distribution of 𝜃 , and P D is a penalty term reflecting the model’s complexity, calculated as: | A lower WAIC value…”

“WAIC was preferred over the Deviance information criterion (DIC) due to its greater reliability in certain circumstances (Gelman, Hwang, and Vehtari 2014).”

“To assess out-of-sample predictive performance, we used the pointwise predictive density (PPD) (Gelman et al., 2014).”

“The AIC was used as a selection criterion with an aim to improve model predictive performance as it serves as an approximation of the out-of-sample predictive error [28,29].”

“To circumvent the problem of choosing K , one can simply make K = N , yielding a special case of K -fold cross-validation called leave-one-out cross-validation (Gelman et al. 2014; Vehtari & Ojanen 2012; Vehtari et al. 2020).”

“Point estimates resulting from cross-validation were used to determine performance instead of Bayesian estimates of out-of-sample generalizability 62,63 to allow this comparison.”

“In applied settings, it is plausible that a single assessment could contain items that function according to different DCMs, forming a mixed DCM (e.g., some items being non-compensatory and others compensatory).”

“We adopt the version of WAIC recommended by Gelman et al. (2014), given by: where the first term is log pointwise predictive density and the second term measures the model complexity by the sample variance.”

“However, the WAIC averages over the posterior distribution rather than conditioning on a point estimate as in the DIC [18].”

“A multilevel model is a regression model that can reflect homogeneity within clusters and heterogeneity between clusters (Gelman, Hwang, and Vehtari 2014).”

“Parameters in a model are not identified if the same likelihood function is obtained for more than one choice of the model parameters (Teicher, 1963; Partial membership models for soft clustering of multivariate football data Gelman et al, 2013).”

“E u r op e P M C F und e r s A u t ho r M a nu s c r i p t s density to correct for the model’s complexity (Gelman et al., 2014).”

“The rank of C was selected from { 1 , 2 , 3 , 4 } based on the Watanabe-Akaike information criterion (WAIC) [19].”

“We assessed model fit by computing WAIC (Gelman et al. 2014, equation 13), and the values are provided in Table 2 for each of the four models considered.”

“…observing 𝒚 𝒊 as belonging to a particular cluster 𝒌 based on (11) (ii) Simulate classification vector values 𝒛 𝒊 from the multinomial distribution determined in the above step (i) 8: In the last step, simulate the latent chain 𝜌 𝑗,𝑘 ’s similarly as in Step 7 from their full conditional…”

“Another variant proposed by Gelman et al. [2014], which is on the same scale as the deviance for ease of comparison, is represented as: In Gelman et al. [2014]’s interpretation, the term p W serves as an approximation of the number of ”free” parameters in the model.”

“However, both rely only on single point estimates, which makes them often mean-ingless in the context of mixture and hierarchical models (Gelman et al., 2014).”

“…prior information (if available), or it can be selected based on information criteria for model assessment, like the ones described by Gelman et al. 29 We have used the R package loo , 30 which uses leave-one-out cross-validation (LOO) and the widely applicable information criterion…”

“Therefore, we decided to report both WAIC and DIC estimates to provide evidence of consensus between the two statistics.”

“Bayesian inference also allows us to integrate the prior knowledge in inference and prediction (Gelman et al., 2014) e.g., the physiological, anatomical, or clinical knowledge to maximize the model predictive power against measurements (Hashemi et al., 2021; Baldy et al., 2023).”

“According to the fact that elpd waic is asymptotically equal to elpd loo [21], the WAIC-based approximate forecasting procedure in the framework of BMA is proposed. π( p | y ) can be approximated by exp { elpd p waic } up to multiplicative constants, leading to the aggregated predictions: where ˆ y…”

“Our focus is on the accuracy of an expert’s forecast density (Gelman et al., 2014).”

“…write: P This sum itself needs to be squared to get the squared error for the population mean P which is why simply summing the squared error of the individuals or cells (as is often done when scoring models; see Gelman et al. 2014) would not provide an acceptable score for the population mean.”

“Model comparison for the two variations of M1 is performed using the Watanabe–Akaike Information Criterion (WAIC) which accounts for both the goodness of fit and the complexity of the model (Gelman et al., 2014).”

“In contrast to BME, ELPD has a smaller influence from the prior, given that it does not play a significant role in posterior predictions when having informative data (Gelman et al., 2014).”

“The DIC is a measure of goodness of fit, penalizing for model complexity [32, 33].”

“…have gained significant attention, particularly in the context of Bayesian model comparison (see, e.g., Marshall and Spiegelhalter 2003, 2007; Gelman, Hwang, and Vehtari 2014; Li et al. 2015; Li, Feng, and Qiu 2017; Piironen and Vehtari 2017; Vehtari, Gelman, and Gabry 2017; Smith, Zheng,…”

“…the goodness of fit of the model and its predictive performance, as well as to select a model from a set of candidate models (Vehtari and Ojanen 2012; Gelman et al. 2014), our approach is quite informal in the sense that the proposed specification is based on the structural knowledge of the…”

“A common utility function that measures the out-of-sample predictive performance of a model is the expected log pointwise predictive density (ELPD) (for details see Gelman et al. (2014) and for practical estimation in a Bayesian framework see Vehtari et al. (2017)).”

“The WAIC value of the proposed joint model is 11945.89 and the sum of the WAIC values of the separate models is 12173.48, indicating that the joint modeling is better than the separate modeling for this data.”

“Similar ideas occur in posterior predictive approaches to model selection, including Bernardo’s Bayesian reference criterion (BRC) (Bernardo, 1999) and Watanabe’s widely applicable information criterion (WAIC) (Gelman et al., 2014; Watanabe, 2010).”

“Inference is drawn from the posterior distribution of the parameters obtained by updating the a priori assumptions via the likelihood function of the data, for an overview see the book by Gelman et al. (2013).”

“…accuracy, we consider the Pareto-smoothed importance sampling-based leave-one-out (LOO) cross validation metric (Vehtari et al., 2015, 2019; Gelman et al., 2014; Vehtari et al., 2017), which approximates the total log likelihood of left out data when fit-ting the model using n-fold cross…”

“HMC outperforms the traditional Gibbs sampler (Geman & Geman, 1984) and the Metropolis algorithm (Metropolis et al., 1953), particularly in terms of computational efficiency in exploring the posterior parameter space (Gelman et al., 2013).”

“The calculation of the WAIC that we use is based on [46], and our notation in this section is identical to theirs.”

“DIC is “a somewhat Bayesian version” of the well-known Akaike information criterion, which measures both the model fit and the complexity (Gelman et al. 2013, 2014).”

“Public Health 2022, 19, 6704 7 of 12 We suggest a Poisson spatial model based on DIC, WAIC, MAPE, and MSE (Table 2), with corresponding values of 12,240, 11,876, 0.035, and 0.002, respectively, with the best goodness of fit.”

“For more robust estimates of the model weights one may also consider replacing the AIC score by other strategies to estimate the cross entropy, such as cross validation, PSIS-LOO or the WAIC score [46, 47].”

“…the expected out-of-sample prediction error of the different functional models using the Watanabe-231 Akaike or other information criteria (Gelman et al. 2014), although the predictive performance of a model may be a poor measure of whether the correct causal relationships has been…”

“…we compare our model selection criteria with information criterion ( i.e. Watanabe–Akaike information criteria (WAIC), expected predictive deviance (EPD), expected Akaike information criterion (EAIC) and expected Bayesian information criterion (EBIC)) [41,42] and demonstrate it works in practice.”

“Both criteria[24] correct for the number of parameters, and smaller values indicate better model fit.”

“Inference for these elements is based on the Watanabe-Akaike information criterion (WAIC); see Gelman et al. (2014). The WAIC uses the posterior sample and an outof-sample test set to estimate the log posterior predictive density with an adjustment for the effective number of parameters.”

“To supplement model selection where we compared goodness of model fit with and without the indicators, we computed fully Bayesian estimates of pointwise predictive density by Watanabe– Akaike information criterion and approximate leave-oneout cross-validation(39,40).”

“…alongside the initial OLS model (Appendix Table 5), we take care not to compare the OLS AIC values with the CAR Bayesian spatial GLM WAIC value, because the latter averages over the posterior distribution rather than conditioning on a point estimate and is thus not analogous or comparable [50].”

“We prioritised WAIC over DIC because it integrates the predictive density over the posterior rather than evaluating fit at a point estimate (Watanabe, 2010; Gelman et al., 2014); the lower DIC of M2-III was therefore not decisive.”

“Convergence of the models was assessed using Gelman–Rubin diagnostic statistics, including the effective sample size (ESS) and the potential scale reduction factor ( ̂𝑅 ) (Gelman et al., 2004, 2014).”

“Depending on the experiment, we evaluate accuracy (ACC), ROC AUC and log pointwise predictive density (LPPD; Gelman et al., 2014) on a large synthetic test set drawn from the TabICL prior or real-world classification datasets from TabArena (Erickson et al., 2025) as well as TALENT (Liu et al.,…”

“The AIC is defined as where L Whereas the AIC penalizes model complexity through the nominal number of free parameters, the deviance information criterion (DIC) [75, 76] provides a posterior-based measure of effective model complexity.”

“The candidate likelihoods and covariate forms are evaluated jointly by two negatively oriented Bayesian criteria: the Deviance Information Criterion (DIC; [41]) and the Watanabe–Akaike Information Criterion (WAIC; [24]).”

“It can provide more accurate estimates of out-of-sample predictive performance (Gelman et al., 2014).”

“Convergence was assessed visually through trace plots and quantitatively using the Gelman–Rubin R̂ statistic (Gelman et al. 2014), with all parameters achieving R̂ = 1.00.”

“Model convergence was evaluated using the potential scale reduction factor ( ), effective sample size (ESS), and posterior predictive checks (Gelman et al., 2014; Vehtari et al., 2021).”

“…the various SBOE functions using WAIC (Watanabe-Akaike Information Criterion), a generalized version of AIC onto singular statistical models, see Gelman et al. (2014) and Watanabe (2013; de Valpine et al., 2020): the mean of ten independent chains was used, each with a length of 50,000 samples…”

“Comparison with other Bayesian malaria modelling studies in other countries identified in the literature may not hence be warranted, since scarce published works have studied survey design under Bayesian spatial models, see [9, 10, 47].”

“…such conditions, the influence of non-informative or flat priors on the posterior distribution asymptotically diminishes as sample size increases (Gelman et al., 2013), making the choice between Mplus’s default priors and mildly informative alternatives unlikely to meaningfully alter the…”