Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“Forward stepwise model selection procedure was applied using uninformative prior hyperparameter estimates for all covariates (n = 10) in separate models for all 15 species [27], to select optimal model structure for each species using the Watanabe– Akaike information criterion (wAIC) [28].”

“Then, following Gelman et al. (2014), the WAIC can be calculated as, where the superscript [ m ] denotes a draw from the posterior of the variable and V ar denotes the sample variance.”

“Instead, the DIC introduced by Spiegelhalter et al. (2002) is often considered to approximate the Bayesian evidence and has successfully been applied in several contexts (Gelman et al. , 2014; Steininger et al. , 2014).”

“SPPD is the same as LPPD (Gelman et al., 2014), but scaled by 1|I| and exponentiated so it is always between 0 and 1, where higher is better.”

“The WAIC can be defined as a deviance scale form by multiplying Equation (23) by−2: WAIC = −2 êlppdwaic = −2 l̂ppd + 2 p̂waic (26) Econometrics 2022, 10, 32 10 of 28 Equation (26) is suitable since it can be compared with other measures of deviance, e.g., AIC and DIC (Gelman et al. 2014).”

“Gelman et al. (2013) Posterior predictive checks can be implemented through a simulation based approach by comparing realized test quantities T ( y , s ) to predictive test quantities T ( y rep , s , s ) , which are based on the S posterior draws.”

“The resulting models are compared using the WAIC criterion (Gelman et al., 2014).”

“Bayesian models can be evaluated and compared in several ways (Gelfand and Dey 1994; Wasserman 2000; Gelman et al. 2014).”

“3 Model Evaluation Measuring predictive accuracy is a way to validate or compare models (Gelman et al., 2014).”

“WAIC estimates expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error, similar to Akaike Information Criterion (AIC) and Deviance Information Criterion (DIC)[22].”

“An intuitive measure of predictive ability of a model is the expected log predictive density (ELPD) for a new data point (Gelman et al., 2014).”

“…1.000280 0.999982 We compare the models with and without the structural account of alternatives with the Watanabe-Akaike Information Criterion (WAIC) [Watanabe, 2013, Gelman et al., 2014] (fig 12), an information criterion that considers the deviance for all posterior samples and is apt for…”

“To this end, we compare different architectures and BNNs in terms of their cumulative log pointwise predictive density (LPPD, Gelman et al., 2014), examining the rate at which this metric saturates (details in Section D).”

“Implementing the U-Score will transform H2F from a binary variable into a precision instrument for intervention optimization and cost-effectiveness mod-eling—aligned with modern implementation science [213, 223].”

“Generally, the model showing lower criterion values fits the data better (Gelman et al., 2014; Levy & Mislevy, 2016).”

“More detailed intro-ductions to model selection are available elsewhere (Gelman, Hwang, and Vehtari 2014; Lartillot and Hohna 2023; W. Wang and Gelman 2014).”

“However, the asymptotic theory that serves as the basis for the AIC does not hold for mixture models (Watanabe 2010; Gelman et al. 2014), and recent work suggests that the AIC is not suitable for assessing mixture models in phylogenetics (Susko and Roger 2020; Crotty and Holland 2022; Liu et al.…”

“For example, see Gelfand and Dey (1994); Wasserman (2000); Gelman et al. (2014).”

“We retain 400 of the 500 available eigenvectors based on minimising the WAIC criterion calculated for each of the 500 possible models including from 1 to 500 eigenvectors of the spatial random effect (see Section 2.4 ).”

“[19] introduce a calculation of log pointwise predictive density in practice.”

“Log predictive density is a common measure of predictive accuracy for methods that output probabilities [Gelman et al., 2014], and expected calibration error measures how well a model’s predictions are calibrated to the class probabilities observed in the data [Guo et al., 2017].”

“The models incorporating the interaction performed significantly better (ΔWAIC = − 79.3 and − 23.8 for activity level and freezing events respectively) than those without.”

“Model comparison was performed using the Waikake-information criterion 42 .”

“The WAIC is used for determining the prediction accuracy of Bayesian models and was introduced by Watanabe (Gelman et al., 2014; Watanabe & Opper, 2010).”

“We then consider the Watanabe–Akaike information criterion (WAIC), proposed by Watanabe (2010), for model selection among different possible models for the data under consideration, see also Gelman et al. (2014), Stander et al. (2019), Watanabe (2013).”

“From a statistical perspective of machine learning models, the evaluation of uncertainty can be regarded as a problem of assessing and replicating the probability distribution pertaining to findings and the physicians’ observations [17, 18].”

“Model fits were compared using the Watanabe–Akaike Information Criterion (WAIC; Gelman et al., 2014; Watanabe, 2010).”

“The fitted models (with and without covariate imputations) were evaluated using the Deviance Information Criteria (DIC) and the Watanabe–Akaike information criterion (WAIC; see [48]).”

“In this case, we set 𝝁 𝜇 = ̂𝛾 , 𝚺 𝜇 = 𝑺 = 𝑛𝑇 ̃𝜎 2 ( 𝑍 ⊤ 𝑍 ) − 1 , and df = 2 ( 𝑞 + 1 ) , where ̂𝛾 and ̂𝜎 2 stand for the ordinary least squares estimates of 𝛾 and 𝜎 2 , respectively.”

“…fit measures are available; the deviance Information Criterion (DIC) (Spiegelhalter et al. 2002), the Wata-nabe-Akaike Information criterion (WAIC) (Watanabe 2010, Gelman et al. 2014) and the log-likelihood (Hubin and Storvik 2016) can be obtained for each of them in order to select the best one.”

“Because there was little knowledge about the impacts of the parameters included, the priors applied in all our models are what are known as weakly informative priors (Carpenter et al., 2017; Gelman et al., 2014).”

“Details on various criteria for comparing models in the Bayesian context are presented by Gelman et al. (2014).”

“Bayesian hierarchical models incorporate the uncertainty around the parameters at the lower levels of the hierarchy and sequentially transfer it to the next; hence, they effectively investigate cross-level hypotheses and minimize the effect of unbalanced data ( Gelman, Hwang & Vehtari, 2014 ).”

“, statistical loss) of making bad predictions (Gelman et al., 2014; Hooten and Hobbs, 2015).”

“For each bias we then compared the predictive power of the three models (including trials c, b&c, a&b&c) using their BIC values (Raftery, 1995; Gelman et al., 2014).”

“Table 3 shows the sum of AIC, DIC, and WAIC for �� distributions over 12 users.”

“The random walk component is not considered in the model for Hennepin county as the estimate of 𝜎 w in (6) is approximately zero, which indicates no significant week effects and the WAICs with and without the random walk appear to be the same.”

“, a model with spatial correlation and a model without spatial correlation) may also be done using deviance information criterion (DIC) [25] and Watanabe–Akaike information criterion (WAIC) [26, 27].”

“There have been two adjustments of WAIC suggested in the literature and both are viewed as the approximations to cross validation [58].”

“The remaining analytes plus a limited number of measurements (lack of one or two preliminary experiments) from the excluded analytes were used to obtain predictions [23, 24].”

“When we compared the model including the effect of humidity with a model excluding it, with the Watanabe–Akaike Information Criterion (WAIC; [69]), we could not detect any significant difference between the two models (model with humidity: WAIC = 1373.”

“We also evaluate Watanabe – Akaike information criteria (WAIC) [16], defined as, where Var i denotes the variance of the likelihood values for observations over posterior samples.”

“However, the BIC is typically inappropriate for hierarchical models since the hierarchical structure implies that the effective number of parameters will typically be lower than the actual number of parameters in the likelihood (Gelman et al., 2014).”

“(8) Two versions of pDIC are generally used (Gelman, Hwang, & Vehtari, 2014; Spiegelhalter, Best, Carlin, & van der Linde, 2014): (i) pDIC in (8) which is considered to be numerically stable, and (ii) pDIC2 = 2 Var𝜽|y[(y|𝜽)] which has the advantage of being always positive (Gelman et al., 2014).”

“We assess the predictive performance using the MAP classifications of out-of-sample observations to components (denoted ˜ k ( n ) = argmax k q (˜ c nk = 1)) and computing the (MAP) negative log point-wise predictive density (Gelman et al. 2014) 3 implemented in Python SciPy (Virtanen et al. 2020).”

“The statistical framework for these models, the link functions and the criteria to select the best performing models is provided in Additional file 3 [33, 34].”

“Following [12], the model selection criteria, AIC, DIC, and WAIC all try to maximize the predictive performance of the model, i.e the accuracy in predicting a new observation.”

“23 According to Spiegelhalter et al., 24 DIC is defined as the difference between the posterior mean of the deviance and the deviance at the posterior means of the parameters of interest.”

“…a fully Bayesian method that estimates out-of-sample predictive accuracy by computing the expected log pointwise posterior predictive density across the posterior distribution and applying a complexity penalty based on the effective number of parameters (Gelman et al., 2014; Vehtari et al., 2017).”

“The five detection models were evaluated using the Watanabe-Akaike Information Criterion (WAIC) 101,102 to identify the most supported detection covariate.”

“Our preferred model was chosen based on: (1) their standardised residuals and (2) their posterior predictive checks on mean and median, as adapted from Gelman 98,99 .”

“The observed differences in DIC (exceeding 5) [75] and WAIC (exceeding 11) are regarded as practically meaningful [84], thereby further supporting the preference for the CAR-ICAR specification.”

“The models were selected according to the posterior predictive distribution and the Widely applicable Bayesian Information Criterion (WBIC) (Gelman et al., 2014).”

“We calculated Watanabe-Akaike Information Criterion (WAIC) [28] for PD ( 𝑊 𝐴𝐼𝐶 = 61582 .”

“32 R-codes for all analyses can be found at https://github. com/KA-asosega/Bayesian-classical-multilevel.”

“…by Handcock et al. [1], we choose the widely applicable information criterion [WAIC,34], since the WAIC averages over the posterior distribution instead of conditioning on a point estimate [35], making it more helpful for models with hierarchical and mixture structures like the proposed CALF-SBM.”

“Model comparisons were made using expected log pointwise predictive density (EPLD) that evaluates Bayesian models based on their predictive performance (Gelman et al. 2014).”

“Consequently, we limited the variability and checked the performance of the Bayesian joint model using the log of the posterior predictive density [18 , 19 ].”