Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“Various approaches exist, including frequentist hypothesis testing, information criteria [3], and cross-validation [4], but Bayesian model comparison is popular in many fields due to its evaluation of relative plausibility and principled penalisation of model complexity [5, 6, 7].”

“As under these circumstances, the DIC approaches the AIC (e.g., Gelman et al. 2013; Spiegelhalter et al. 2002), to denote the averaged imputed dataset, the pooled parameters – pooled datasets AIC , AIC PPD , is given by (5) Following the above, the pooled estimator for the BIC , which we will refer…”

“First, a Bayesian sensitivity analysis across Non-informative, Weakly-informative, and Robust informative priors produced ΔWAIC ≤ ~0.4 with consistent coefficient directions, indicating robustness to prior choice (Gelman et al., 2013; Gelman, Hwang, & Vehtari, 2014; Bürkner, 2017).”

“Furthermore, it is invariant under parametrizations [10, 24, 30].”

“In the Bayesian framework, model selection can be performed using a variety of techniques which are reviewed and compared in [21] and [9].”

“29 For the cross-validation, the dataset was randomly split into 10-folds, and for each candidate number of subtypes, the model was repeatedly trained on 9-folds and evaluated on the remaining fold to obtain out-of-sample log-likelihoods.”

“…non-centred hierarchical TABLE 1 Characteristics of Mugil cephalus and Mugil curema specimens: locations, group code, number of specimens ( N ), number of scales ( n ), sex, collection date and standard length (cm). hypotheses and minimize the effect of data imbalances (Gelman et al., 2014).”

“V.”

“Then, X j is divided into a j � 1 equal sub-intervals as ½ x 0 j , x 1 j � , ½ x 1 j , x 2 j � , ::: , ½ x The number of sub-intervals is determined by the Deviance Information Criterion (Gelman et al., 2014), which has been widely used to balance the fitting ability and complexity of Bayesian model.”

“For more information on WAIC computation, see [21].”

“Furthermore, multiple versions of DIC have been introduced in the literature [94], particularly to address challenges in hierarchical models; however, in this study, we adopt the standard form of DIC as originally proposed by Spiegelhalter et al. [92].”

“The three spatial structures for the spatiotemporal Poisson model presented in section 2, namely ICAR, PCAR, and LCAR, are compared using the Watanabe-Akaike information criterion (WAIC hereinafter, Gelman et al., 2014), computed internally in R-INLA .”

“…or martingale-like structure, and it has since become conventional wisdom that CV will work well if this is the case (see for example Burman et al. (1994), Racine (2000), Arlot and Celisse (2010), Gelman et al. (2014), Bergmeir et al. (2018), Coulombe et al. (2022), Petropoulos et al. (2022)).”

“A cross-validation technique is used to choose the appropriate c that results in the smallest WAIC (Gelman et al., 2014) evaluated only by the non-zero part of the data used in the modeling due to variation in the sample size for different values of c.”

“26 We will rely on leave- one-out cross-validation 27 28 and posterior predictive checking for model-fitting.”

“This yields more reliable diagnostics, as it breaks the circularity of using the same data for both model fitting and evaluation [49, 15, 10].”

“The WAIC computes the logarithm of the pointwise posterior predictive density (LPPD) and adds the penalty term to adjust for overfitting (Gelman et al. 2014).”

“For model comparison and weight we used the Watanabe-Akaike Information Criterion (WAIC; Watanabe & Opper, 2010), a generalized version of the Akaike Information Criterion (Gelman et al. , 2014).”

“Compared to the AIC (Akaike 1974) and deviance information criterion (DIC) (Spiegelhalter et al. 2002), WAIC averages over the posterior distribution rather than using a point estimate directly and thus is considered more suitable for a Bayesian context (Gelman et al. 2014).”

“…e.g., Gamerman and Lopes 2006) methods, specifically the Metropolis–Hastings algorithm (e.g., Albert et al. 2009), the Hamiltonian Monte Carlo algorithm (e.g., Gelman et al. 2013a), and the Spherical Hamiltonian Monte Carlo algorithm (e.g., Yu 2020), which are developed throughout this section.”

“This yields “ Bayesian hypothesis tests ” at a two-sided signi fi cance level of 0.05 (Gelman et al., 2014).”

“Methods to estimate the effective number of parameter in the DIC can be numerically unstable, undermining their applicability (Gelman, Hwang, & Vehtari 2014).”

“Further tests of how good of a fit each function is are the Bayesian Information Criterion (BIC), Akaike Information Criterion (AIC) [11], and the minimum message length (MML) [31].”

“Bayesian inference is carried out by MCMC, where we sample from the posterior distribution by the Hamiltonian Monte Carlo algorithm (Gelman et al. 2013).”

“The predictive accuracy for each model is measured using the widely applicable information criterion, WAIC (Watanabe, 2010; Gelman et al., 2013).”

“In computing the WAIC, we follow the second approach proposed by Gelman et al. (2014), which estimates the effective number of parameters using the variance of the log predictive density.”

“Following Gelman et al. (2014), Burnham and Anderson (2004), and Vehtari et al. (2017), WAIC differences below five are not considered meaningful because they may arise from small fluctuations in the data rather than true model differences.”

“Comparisons between PUM and other Bayesian models can be carried out using a blocked version of the Watanabe-Akaike Information Criterion (WAIC, Watanabe and Opper 2010; Watanabe 2013; Gelman, Hwang, and Vehtari 2014).”

“A fitted model with the highest expected log-predictive density carries the highest posterior probability [31].”

“The ratio r is selected by minimizing Watanabe-Akaike Information Cr iter ion (WAIC) of the outcome model (Gelman et al., 2014 ).”

“Among the 15 possible models, we selected the one with posterior convergence (i.e. R-hat < 1.1 and n.eff > 60 000) and the lowest Widely Applicable Information Criterion (WAIC) (Watanabe 2010, 2021; Gelman et al. 2014).”

“Another possible option would be using WAIC (Watanabe , 2010) as a comparison metric, but since WAIC has being reported to have issues with correlated data (Gelman, Hwang & Vehtari , 2013), such as spatial or temporal data, we have opted for using DIC.”

“We would also like to note that an improper uniform is not considered non-informative in the strictest sense (see e.g. Gelman et al., 2014; Jeffreys and Swirles, 1972).”

“During the feature selection process, the Bayesian Information Criterion (BIC) was used to select the final features of the model, penalizing the addition of new terms (Gelman et al., 2014).”

“19 Vehtari et al 20 propose approximating this quantity with a leave-one-out formulation (ELPD loo ), also known as the “log pseudo-marginal likelihood” (LPML), 21 which sums over the log posterior predictive distributions for each observed data point of a model fit excluding that data point…”

“To this scope, we introduce a variational version of the Watanabe-Akaike information criterion (WAIC; Watanabe, 2010; Gelman et al., 2014) and also consider the variational Akaike information criterion (VAIC) proposed by You et al. (2014).”

“The two criteria also differ in that AIC is calibrated for improving predictive accuracy, while may be viewed as favoring the model that best approximates the marginal distribution of the data [Gelman et al., 2014].”

“We assessed relative model fit by means of the Watanbe–Akaike information criterion (WAIC; also known as the ‘widely applicable information criterion’), which is fully Bayesian in that it is obtained from the posterior distribution rather than from a point estimate [30].”

“A escolha do método bayesiano justifica-se devido ao fato de que métodos de máxima verossimilhança podem, em modelos com número elevado de variáveis, ser um limitador (Gelman et al., 2014).”

“17 However, the computational challenges and convergence issues associated with Markov chain Monte Carlo (MCMC) techniques pose limitations.”

“Using this model, we demonstrated how to perform model selection for the inclusion of covariates using WAIC.”

“The eldp is an indicator quantifying the predictive accuracy of a model on the data it was trained on [37].”

“As introduced by Gelman et al. (2014), this method offers a distinct approach to assessing model performance.”

“The AIC and Bayesian variants [Gelman et al., 2014, Watanabe, 2013, Vehtari et al., 2017, Piironen and Vehtari, 2017a] are asymptotic approximations [Stone, 1977, Watanabe, 2010] of a model’s leave one out (LOO) cross validation loss.”

“The out-of-sample predictive accuracy — or predictive ability — of an expert has traditionally been treated as a fixed, unknown quantity (Gelman et al., 2014).”

“For this, following Gelman et al. (2014) and Wiese et al. (2023), we choose the log posterior predictive density (LPPD) over a test set, defined as The LPPD quantifies how well, on average, the predictive distribution covers the observed label.”

“Note that by identifying the points y i that are sensitive to the certain posterior samples estimation, the WAIC allows for assessing overfitting at the level of individual observation [16].”

“The DIC was used to evaluate model fit as an approximate to the AIC [29].”

“However, there are other measures for model assessment and selection (see Gelman et al. (2014) and Vehtari and Ojanen (2012)), such as the expected log predictive density, E{log π(y|z)|y, η}, and the widely applicable information criterion (WAIC).”

“In the Bayesian framework, model selection can be performed using a variety of techniques which are reviewed and compared in O’Hara and Sillanp¨a¨a (2009); Gelman et al. (2014).”

“26 Model generation Stan 20 is a programming language for specifying statistical models and used as a Markov chain Monte Carlo sampler for Bayesian analyses.”

“For model selection, we determined the AIC and BIC index (Gelman et al., 2014).”

“We estimated three models with different combinations of covariates Cµ, Cγ, Cπ, all suggested by clinical evidence, and among them, we selected the best model using the WAIC[11] criterion (see Table 3 ).”

“The widely applicable information criterion (WAIC) statistic was calculated using ˆelppdWAIC = lppd − pWAIC, where lppd = log ∏n i=1 ppost(yi) = ∑n i=1 log ∫ p(yi | θ)p(θ | y)dθ is the log pointwise predictive density. lppd is computed using ∑n i=1 log ( 1 L ∑L l=1 p(yi | θl) ) and the correction term pWAIC = ∑n i=1 V…”

“WAIC is obtained as W AIC = −2(lppd − pwaic) where lppd refers to log pointwise predictive density which measures the predictive accuracy of the fitted model [23].”

“The optimal number of subtypes was determined using information criteria calculated through cross-validation 81 to balance model complexity with internal model accuracy, as described previously.”

“Gelman et al. (2014) discuss the limitations of relying solely on log predictive density methods (such as WAIC) for model selection.”

“In addition to DIC, WAIC introduced by Watanabe has the preferable property by averaging over the posterior distribution instead of conditioning on a point estimate.(46) As with Akaike information criterion (AIC) or Bayesian information criterion (BIC) used in a frequentist framework, models with smaller DIC, DIC3, or…”

“Four versions of deviance information criterion were used to compare model fit: DIC, DICalt, and the fully Bayesian versions WAIC1 and WAIC2 (Watanabe, 2013; Gelman et al., 2014).”

“The second is the log pseudo marginal likelihood (LPML, [15, 16]).”