Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“In practice, multiple chains are initialized from dispersed starting values, and an initial warm-up (burn-in) segment is discarded before retaining draws for inference (Neal 1993; Gelman et al. 2013).”

“These checks indicate whether the fitted models can reproduce the main features of the observed data rather than measuring parameter uncertainty (Gelman et al. 2014).”

“Model performance was assessed using leave one out cross validation (LOOIC), which balances model complexity with model fit (Gelman, Hwang, & Vehtari, 2014) and change in the difference in expected log point wise predictive density from the best model ( (cid:2) ELPD).”

“Model adequacy was evaluated through posterior predictive checks (Gelman et al., 2013) .”

“As discussed in the literature (Gelman et al., 2014), the Watanabe-Akaike information criterion (WAIC; Watanabe, 2010) is one of the most common and best performing metrics for model selection.”

“Model predictions were evaluated using the log pointwise predictive density (lppd) score averaged across cross-validation folds (Gelman et al., 2014).”

“Together, WAIC and LOO provide complementary assessments of model fit and predictive accuracy, enhancing the robustness of our findings on DSR-related nonlinearities.”

“We considered the model with the lowest WAIC score to be the best performing and kept this constant for all further analyses (Watanabe 2010; Gelman et al. 2013; Table S1).”

“The definitions and formulations presented in this subsection are adapted from Gelman et al. [11], but the notation follows Section 3.”

“…applicable information criterion (WAIC), which provides an estimate of out-of-sample predictive accuracy while accounting for model complexity (Gelman et al. 2014; Watanabe 2013); lower values of WAIC indicate improved predictive performance with differences in WAIC of more than 5–10…”

“Maximizing the ELPD amounts to minimizing the Kullback-Leibler divergence between the true generative distribution of the data and the predictive distribution of the model [26].”

“This criterion provides an estimate of out-of-sample predictive precision by integrating across the entire posterior distribution (Gelman et al., 2014).”

“The Bayesian Information Criterion (BIC) estimates the maximal observed log likelihood and adds a penalty for the number of model parameters, to assess how well a candidate model fits the observed data on which the models were trained [Gelman et al., 2014].”

“…the same role as the effective parameter counts of the Bayesian model-comparison literature, p D in the DIC (Spiegelhalter et al., 2002) and p WAIC (Gelman et al., 2014; Watanabe, 2010), and as the generalized degrees of freedom used to price data-driven selection in regularized regression (Efron,…”

“In INLA it is defined as (Gelman, Hwang, & Vehtari, 2014): Effective number of parameter , where θ represents the posterior distribution of the model and s are samples.”

“In Bayesian model selection, lower values of LOOIC and WAIC indicate better model fit, while higher values of elpd _ loo and P _ waic suggest better predictive accuracy and greater model complexity (Burnham and Anderson, 2002; Vehtari et al., 2017; Gelman et al., 2014).”

“To compare these two alternatives, we fixed the corresponding model parameter and compared the resulting Watanabe-Akaike Information Criterion (WAIC), a measure of model parsimony [26] that penalizes models for their complexity.”

“* This is a WAIC model comparison for models fitted with computational Bayesian simulation 29 . of 27 ᵒ C and rainfall of 2,631 mm 30–32 ; and marked seasonality, where the dry season extends from December through April.”

“Based on Bayes’ theorem, the incorporation of prior information, which comes from background knowledge, expert opinion, previous research, or personal beliefs, coupled with observed sample data produces posterior distribution of the parameters (Gelman et al., 2014).”

“The optimal mesh configuration for each colony and census was selected using the Watanabe-Akaike Information Criterion (WAIC, Watanabe 2010), which assesses model predictive performance while penalizing model complexity (Hoeting et al. 2006; Gelman et al. 2014).”

“WAIC is a fully Bayesian information criterion based on the pointwise posterior predictive density and is widely used for hierarchical Bayesian models (Gelman et al. 2014).”

“A fully Bayesian method such as WAIC (Watanabe and Opper, 2010; Gelman et al., 2014) could be an alternative approach, but its implementation based on the MCMC approach of Algorithm 1 would require conditional simulations.”

“…Criterion (WAIC) is based on pointwise predictive densities adjusted for the effective number of parameters, and because it is asymptotically equivalent to Bayesian leave-one-out cross-validation, lower WAIC values indicate better expected predictive ability (Watanabe 2010; Gelman et al. 2014).”

“In addition, the Widely Applicable Information Criterion (WAIC; Watanabe, 2010) may be used as an alternative fully Bayesian measure that accounts for the entire posterior distribution rather than relying on point estimates (Gelman et al., 2014).”

“The LVGP-Sym model performed worse ( Δ ELPD WAIC = − 995.3, comparing LVGP-Sym ELPD WAIC = − 2982.41 relative to the LVGP-Asym ELPD WAIC = − 1987.04), ranking last among policy models (see Supplementary Table S1 and Fig.”

“2 In addition, we report the minimum and maximum runtime over the 10 tests and the log posterior predictive density (LPPD; Gelman et al., 2014) Together, these metrics separate latent structure recovery from predictive fit.”

“A more complete and principled choice is the negative log predictive density (NLPD) [16], defined as: which evaluates both the accuracy of the mean prediction and the calibration of the associated uncertainty, penalizing both overconfident and underconfident predictions.”

“We utilize the WAIC because it is a more stable cross-model evaluator due to its pointwise variance calculation (Gelman et al., 2014).”

“Predictive model-evaluation research contributed to the comparison by showing how predictive criteria can assess Bayesian model quality beyond point estimates alone [9].”

“Equation (3) uses the exact Dirichlet posterior variance Var θ | n j [log θ c ] = ψ ′ ( n jc + α ) − ψ ′ ( N j + α 0 ) (Gelman et al., 2014), avoiding Monte Carlo.”

“The optimal number of subtypes was determined as two based on the lowest Cross-Validation Information Criterion (CVIC = 10,998.33; Supplementary Figure 1).”

“…prior combinations and selected the one with the lowest Watanabe-Akaike Information Criterion (WAIC) score (Watanabe, 2010), which estimates out-of-sample predictive accuracy by integrating over the full posterior distribution, appropriately accounting for model complexity (Gelman et al., 2013).”

“WAIC and LOO-CV can be viewed as the criteria of choice for evaluating predictive accuracy of Bayesian models (e.g., Gelman et al., 2014b; Vehtari, Gelman, and Gabry, 2016) because they are based on fully Bayesian posterior predictive densities.”

“Unlike the Deviance Information Criterion (DIC) or the Akaike Information Criterion (AIC), WAIC utilizes the entire posterior distribution rather than a single point estimate, making it more robust for hierarchical models with complex parameter spaces 27 .”

“To put elppd on the familiar deviance scale (as expressed by information criteria), we multiplied LOO-CV elppd values by negative two (Gelman et al. , 2013).”

“With no ground-truth partial order, we report list-level WAIC [38, 10], runtime, and threshold-free posterior comparisons.”

“Posterior distributions were obtained through three Markov Chain Monte Carlo algorithms ran until convergence (i.e., potential-scale reduction factors < 1.01 and effective sample sizes > 2000 for all parameters; Gelman et al. 2014 ; van de Schoot et al. 2021; Supplementary Information).”

“…the pointwise predictive probability function p ( y i ) assumes the form of a pointwise posterior predictive probability (density) function (Gelman et al., 2014), p k ( y i | θ ) p ( θ | y ) dθ , i.e., the posterior predictive probability (density) of each data point y i , after the model…”

“21 The simplest model used was the Fixed Effects Model, where the covariates are the only parameters estimating the SPR.”

“We compare the Deviance Information Criterion (Gelman et al. 2014) of the correctly specified Generalized Poisson models and the misspecified Poisson models under the three specifications M 1 , M 2 , and M 3 .”

“The Figure 2: Random Forest feature importance for predicting unsupervised cluster labels under LOOCV.”

“In addition, we used a leave-one-out approximation from the ‘brms’ package to check the diagnostics of our models (Gelman et al. 2014 ; Vehtari et al. 2017 ).”

“Model comparison was performed using the Expected Log Pointwise Predictive Density (ELPD; Gelman et al. (2014)), estimated via Leave-One-Out Cross-Validation (LOO-CV) using Pareto-Smoothed Importance Sampling (PSIS-LOO; Vehtari et al. (2024)).”

“The selected model was the one with the lowest WAIC among the models explored during this sequential process [37].”

“We compute it using the R package loo [Gelman et al., 2013b, Vehtari et al., 2017].”

“…of INLA is its ability to compute model comparison criteria, such as deviance information criterion (DIC 39 ) and Watanabe-Akaike information criterion (WAIC 40,41 ), and various predictive measures, for example, conditional predictive ordinate (CPO 42 ) and log-score (LS 43 ) to compare the…”

“…the burn-in period (5,000 and 10,000, respectively) were determined from a prelim-inary simulation that collected the potential scale reduction (Gelman & Rubin, 1992) and number of effective MCMC samples (Gelman et al., 2014) diagnostics from a small number of replications in selected design cells.”

“In hindsight, the poor performance of DIC and WAIC is probably to be expected (see e.g. the discussion of Section 7.3 of [6], and Section SM1.”

“Standard criteria developed to select an adequate level of detail of the data representation often rely on supervised information about the target distribution (e.g., likelihood optimisation or divergence minimisation) [14–16], but such an approach is naturally excluded in unsupervised settings.”

“The CV score for a model was quantified using the log predictive density (Gelman et al., 2014; Vehtari et al., 2017).”

“To select the best-fitting model, the deviance information criterion (DIC) and the Watanabe-Akaike information criterion (WAIC) were employed to compare the merits and demerits of the models [30–32].”

“There has long been a predominant focus on conventional model comparison and selection procedures for PCMs, particularly those based on information criteria applied using maximum likelihood ( Goloboff & Arias, 2019 ) or Bayesian inference ( Gelman et al., 2014 ; Holder & Lewis, 2003 ).”

“Convergence was assessed with trace plots and Gelman–Rubin statistics (all ≤ 1.1) (Gelman et al. 2014).”

“…the fits of the competing birth cohort models in equations (1) to (3) (and those of MODsmooth and MODall), we employed the Watanabe-Akaike Information Criterion (WAIC) [41,42] which uses the log of the predictive density of each data point to evaluate a model’s out-of-sample predictive performance.”

“To compare base we calculated Watanabe-Akaike information criterion (WAIC), deviance information criterion (DIC), and mean logarithmic conditional predictive ordinate (LCPO), with lower values indicating better model fit [59,60].”