Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“Several other approaches to quantitative model comparison have been proposed (Gelman et al., 2014; Piironen et al., 2017).”

“We start our analysis by investigating model fit using the Watanabe-Akaike Information Criterion (WAIC, Watanabe and Opper, 2010, Watanabe, 2013, Gelman et al., 2014).”

“We will show below (Thm. 1) that the expected log predicted density [10] ELPD = E log Pr( t | ϕ ) of the classifier on validation data is a lower bound to a divergence D 1 between p ( θ | y ) and q ( θ | y ) up to the known constant h ( w ) , where w = 1 / ( M + 1) and r ( θ | y ) = wp ( θ | y ) +…”

“Data were analysed using Bayesian mixed models (Gelman et al., 2014; McElreath, 2016) with accurate parade identifications scored as 1 and inaccurate identifications as 0, in a 3 (sample duration: 15, 30, 60 s) × 2 (target presence: present or absent) factorial design.”

“Accessible discussions regarding WAIC are provided by Gelman et al. (2014) and Sahu (2022).”

“…reproduce both asexually or sexually, and facultative sexual reproducing species typically engage in sexual reproduction only infrequently and when faced with adverse conditions (e.g. Hairston and Olds, 1984; Simon et al., 2002; Stelzer and Snell, 2003; Lynn and Doerder, 2012; Gerber et al., 2018).”

“196 197 We used a WAIC (widely applicable information criterion; Gelman et al. 2014) score, which is an 198 appropriate criterion for Bayesian applications, to evaluate support for each model and then compared the 199 posterior probabilities of the most supported model to test for impacts of the…”

“This sampler has been chosen because it automates the tuning of the algorithm parameters such as the accuracy, and because it leads to faster convergence and a reduction of correlation in the Markov chains (Hoffman & Gelman, 2014).”

“The WAIC approximates cross-validation and is calculated using the computed log pointwise predictive density lppd = ∑n i=1 log{ 1 M ∑M m=1 p(yi|θm)}, as well as a complexity penalty pWAIC = ∑n i=1 Var[log{p(yi|θ)}] (see Gelman et al., 2014, for more discussion).”

“By default, it divides each chain in half to identify non-stationary chains (Gelman et al., 2013; Carpenter et al., 2017 parameters estimated over simulations and changes within a reasonable tolerance level (not shown here) showed a well-mixed chain.”

“Each competing model was ranked from lowest to highest using the Widely Applicable Information Criterion (WAIC, Watanabe, 2010) and the Akaike weight of each model (Gelman et al., 2014; McElreath, 2020).”

“WAIC is reported on the deviance scale (Gelman et al. 2014) hence a difference in WAIC value of 2 to 6 is considered positive evidence, a difference of 6 to 10 strong evidence, and a difference >10 very strong evidence for one model over another (Kass and Raftery 1995).”

“Vehtari et al. (2017) recommend the use of LOO and WAIC because they have “the desirable property of averaging over the posterior distribution rather than conditioning on a point estimate” (Gelman, Hwang, et al., 2014, p. 1003).”

“These stronger constraints reduce parameter degeneracy and limit the set of circuit solutions consistent with the measured spectrum 32,33 , leading to a narrower predictive error band and more stable inference toward lower frequencies.”

“…is a fully Bayesian method that leverages the whole posterior distribution, as opposed to other metrics such as the Deviance Information Criterion (DIC) (Spiegelhalter et al., 2002), which is less preferable when the posterior distribution is not well summarized by its mean (Gelman et al., 2014).”

“…including data science, physics, engineering, and Bayesian statistics, where high-dimensional integrals arise in tasks such as model selection (Gelman, Hwang, and Vehtari, 2014; Goggans, Henderson, and Xiang, 2013), parameter estimation and uncertainty quantification (Andrieu et al. , 2003;…”

“In this network, latent space models primarily capture degree and geodesic distance patterns, as evidenced by the posterior predictive p -values and the inclusion of observed statistics within the posterior predictive intervals (Gelman et al ., 2014).”

“First, we make use of the Deviance Information Criterion (DIC) [63–67].”

“This reflects a structural limitation of information criteria in hierarchical models: the log-likelihood improvement from finer spatial structure scales with 1 /d i , while penalty terms remain bounded by the number of areas m (Gelman et al., 2014).”

“We assessed subtyping solutions with 2 – 5 clusters, and an optimal choice of 3 was chosen according to its lower CVIC 88 and higher log-likelihood through ten-fold cross-validation (Supplementary Fig.”

“This quantity measures predictive variance across the posterior and acts as an effective complexity correction in WAIC [8, 3].”

“The Deviance Information Criterion (DIC) is a model selection metric that provides a balance between goodness of fit and model parsimony, assessing how well a model proposed predicts new data [9, 31].”

“Model 13 introduced an interaction term between population density and rodent habitat suitability in the fixed effects, but demonstrated little difference from Model 8 in WAIC and DIC, implying that a linear additive structure already captured the bulk of the interaction effect.”

“The predictive accuracy for each model is measured using the widely applicable information criterion, WAIC (Watanabe 2010; Gelman et al. 2013).”

“The correlation term is easily interpretable and is a common goodness-of-fitstatisticusedinmanysettings,includingBayesian spatiotemporal modeling (Gelman et al. 2014).”

“WAIC is considered a better estimate of out-of-sample deviance than the deviance information criterion (DIC; Gelman et al. 2013).”

“Model comparison through leave-one out criterion [84] suggested that natural log transformation of response and independent variables provided regression models that best fit observed data (model comparisons show in supplementary information) in both regression models.”

“Models were compared using the Watanabe Akaike Information Criteria (WAIC, Gelman et al. 2014); a reduction of 2 or more in WAIC was used to select the best fitting model.”

“Markov chain Monte Carlo simulation was performed using the No-U-Turn Sampler (NUTS; Hoffman and Gelman 2014).”

“Here we give examples from two similar data sets where we show how the user could fit multiple functional forms and choose between them using information criteria, specifically WAIC (widely-applicable information criterion, [15]).”

“The approximate leave-one-out cross-validation (LOO) and Watanabe information criterion (WAIC) are used to compare model fit [21].”

“We used the expected log pointwise predictive density (elpd) to rank the models, which reflects how the model predicts future data, assuming that it came from the same distribution as the observed data (Gelman et al. 2014).”

“The winning model was determined using the Widely Applicable Information Criterion (WAIC) 38,39 and a posterior predictive check which demonstrated that a model with two learning rates dependent on stimulus trial type and one outcome sensitivity parameter effectively captured choice accuracy…”

“In that case, expected log predictive density (Gelman, Hwang, and Vehtari 2014) is a commonly used loss function useful for scoring the accuracy of the prediction.”

“However, in real-world applications, 𝑆 is usually unknown and should be determined in a data-driven fashion [26] reviewed various model selection criteria for evaluating Bayesian models.”

“ELPD and LOOIC are commonly used model comparison statistics for Bayesian modelling and are analogous to AIC, such that they can appropriately account for different degrees of freedom among competing models (Gelman et al. 2014).”

“To this end, define the statistical test log-likelihood (TLL) error score as [34, 35]: i.e. a sample mean of logarithmic RPD density values evaluated at test outputs.”

“Under the Bayesian approach, we consider the deviance information criterion (DIC) proposed by Spiegelhalter et al. [32], the expected AIC (EAIC) and its Bayesian version (EBIC); see, for example, Gelman et al. [17].”

“It offers a robust assessment of a model’s predictive accuracy while mitigating the risk of overfitting making it particularly suitable for complex hierarchical models (Gelman et al. , 2014).”

“However, AIC and BIC rely solely on point estimates and do not account for uncertainty in parameter estimation (Gelman et al., 2014).”

“We confirmed convergence by assessing whether the values of all estimates were < 1.1 (Gelman et al. 2014).”

“There has long been a predominant focus on conventional model comparison and selection procedures for PCMs, particularly those based on information criteria applied using maximum likelihood (Goloboff and Arias 2019) or Bayesian inference (Holder and Lewis 2003; Gelman et al. 2014).”

“Model convergence was assessed by checking that the Rhat statistics were equal to 1.0 and by visually examining the traceplots (Gelman et al. 2014).”

“Since ^ lppd is biased, adding a bias reduction term p WAIC to the estimator ^ lppd , the improved estimator WAIC is obtained as: where The WAIC criteria is an approximation to the LOO cross-validation (Gelman, Hwang, and Vehtari (2014)).”

“…any convergence issues, accordingly, we found that the MCMC showed stationarity and convergence to a common target, while Rhat values were all below 1.01, suggesting that different chains came to the same conclusion and that there were no divergent transitions after warm-up (Gelman et al. 2014).”

“These include marginal likelihood methods (Bernardo and Smith 2000) and information criteria (Gelman, Hwang, and Vehtari 2014; Vehtari, Gelman, and Gabry 2017).”

“A point estimate of the underlap coefficient can be obtained by taking the mean or median of the ensemble of posterior realisations credible interval can also be derived from the 100( α/ 2)% and 100(1 − α/ 2)% percentiles of the same ensemble.”

“The effective number of parameters p W is commonly estimated using the variance of the log pointwise predictive density over all subjects as follows: This measure of ˆ p W is most common and provides results closer to LOO (Gelman et al., 2014).”

“In a survey of predictive information criteria Gelman et al. (2014) note that p W gives a better approximation to leave-one-out cross-validation error than p ∗ W .”

“…criteria, such as deviance information criterion (DIC; Spiegelhalter et al., 2002) and Watanabe-Akaike information criterion (WAIC; Watanabe, 2013; Gelman et al., 2014), as well as various predictive measures, e.g., conditional predictive ordinate (CPO; Pettit, 1990) and Log-score (LS; Adrion and…”

“For each MAP estimate, we select the learning rate that results in the SWAG approximation with the highest validation ELPD as well as the learning rate that results in the SWA solution, θ SWA , with the highest validation ELPD.”

“…of the model, (cid:100) elpd loo was shown to be more robust than (cid:100) elpd WAIC ; in the presence of limited sample size and weak priors, WAIC can severely underestimate ˆ p WAIC and often has a larger bias towards the log predictive density [Vehtari et al., 2016, Gelman et al., 2014].”

“The model with lower DIC/WAIC yields a better fit for the data [56].”

“The p WAIC defined here corresponds to p WAIC2 in Gelman et al. (2014), which is recommended because its results are closer in practice to the results of leave-one-out cross-validation.”

“After we confirmed the accuracy of the MCMC iterations, all results showed convergence, lacked divergent transitions following warm-up, and had Rhat values below 1.01 (Gelman et al. 2013, 2014) and determined the credible intervals and mean estimates of marginal posterior distributions.”

“…(2002); Expected Akaike Information Criteria (EAIC), Expected Schwarz-Bayesian (EBIC) and Watanabe-Akaike Information Criterion (WAIC), presented in Gelman et al. (2014); the Bayesian predictive Information Criterion (IC) and Hannan-Quinn Information Criterion (HQIC), presented in Burnham and…”