Understanding predictive information criteria for Bayesian models

“There is also something called the Bayesian information criterion (a misleading name, we believe) that adjusts for the number of fitted parameters with a penalty that increases with the sample size, n (Schwarz 1978).”

“The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure (see Burnham and Anderson, 2002, and Robert, 1996).”

“Examples include the quadratic, logarithmic, and zero-one scores, whose properties are reviewed by Gneiting and Raftery (2007).”

“In this asymptotic limit, the posterior is dominated by the likelihood—the prior contributes only one factor, while the likelihood contributes n factors, one for each data point—and so the likelihood function also approaches the same normal distribution.”

“When several candidate models are available, they can be compared and averaged using Bayes factors (which is equivalent to embedding them in a larger discrete model) or some more practical approximate procedure (Hoeting et al., 1999) or continuous model expansion (Draper, 1999).”

“For practical reasons alone, there remains a place for simple bias corrections such as AIC (Akaike, 1973), DIC (Spiegelhalter et al., 2002, van der Linde, 2005), and, more recently, WAIC (Watanabe, 2010), and all these can be viewed as approximations to different versions of cross-validation (Stone,…”

“Bayesian cross-validation works also with singular models, and Bayesian LOO-CV has been proven to asymptotically equal to WAIC (Watanabe, 2010).”

“As is well known in hierarchical modeling (see, e.g., Spiegelhalter et al. 2002; Gelman et al. 2003), the line separating prior distribution from likelihood is somewhat arbitrary and is related to the question of what aspects of the data will be changed in hypothetical replications.”

“For evaluating the quality of the full predictive distribution and uncertainty, we utilize the log pointwise predictive density (LPPD) on a held-out test set D test , as advocated by Gelman et al. (2014).”

“The model was compared with the original one using the deviance and Watan-abe–Akaike’s information criteria, and log pseudo-marginal likelihood [38].”

“And finally, because the main interest was identifying different species with shared or unshared responses to sea ice (e.g., same or different regression coefficients across species), we used WAIC-based model selection (Gelman et al. 2014; Watanabe 2010) only in a reduced model set based on covariates with higher indi…”

“Within each set of 512 models, individual models were then evaluated and compared using leave- one-out cross-validation ( LOO , Gelman et al. 2014; Vehtari et al. 2017).”

“Posterior distributions were generated for all parameters, accounting for both site-and species-specific variations, providing mean estimates, standard deviations, and 95% credible intervals, quantifying uncertainty in estimates [60].”

“…given as µ and µ + µ 2 φ and the Models incorporating different covariate combinations were fitted, and their relative performance was assessed using the expected log predictive density (elpd) from leave-one-out (LOO) cross-validation [20] and the Watanabe-Akaike information criterion (WAIC) [21].”

“To assess model convergence, we used the Gelman-Ruben statistic (where values < 1.1 indicated convergence across all three chains for each estimated parameter, Gelman et al. 2014) and visual examination of the chains through trace plots (Fig.”

“Additionally, there are numerous criterion to choose from with various penalty terms (Gelman et al., 2013), further complicating the multimodel inference that is required when implementing the two-step and finite-mixture based approaches.”

“Table 6 also gives their respective expected log pointwise predictive density (LPD) [144] as evaluated in Pareto-smoothed importance sampling leave-one-out cross-validation [145, 146], even though a reliable objective evaluation is limited due to the lack of data to compare predictions against.”

“The WAIC is similar to the Akaike information criteria as it estimates the out-of-sample predictive accuracy using a combination of the in-sample log-likelihood and a punishment for the degrees of freedom of the model 46,47 .”

“…could theoretically be achieved with, for example, maximum likelihood approaches, too, Bayesian inference quantifies uncertainty in parameters estimates more accurately and also uncertainty in model comparison (LOO-CV) is accounted for through posterior distribution samples (Gelman et al., 2014).”

“The same folds are used for all the models under study, and the log pointwise predictive density (Gelman et al. 2014, Nicenboim et al. 2021) was used to measure the cross-validation’s predictive accuracy.”

“Models were compared using the deviance information criterion (DIC) and the Watanabe–Akaike information criterion (WAIC), and the significance of variables was indicated by whether the 95% highest posterior density intervals included zero (Cremers, 2021; Gelman et al., 2014).”

“Although the Watanabe-Akaike information criterion (WAIC) is considered superior to the DIC as it uses the entire posterior distribution, it is not recommended for models where there is dependence between observations [72].”

“The overall goodness-of-fit is measured via Watanabe-Akaike Information Criterion (WAIC, Watanabe and Opper [2010]), following the good practice of Gelman et al. [2014] whenever finite mixtures are fitted. We also report the overlapping index (OV , Pastore [2018], Pastore and Calcagnì [2019]) between the posterior dis…”

“This can be done by introducing a scoring rule that is referred to as the expected log pointwise predictive density (elpd) [49], which measures the quality of a model’s fit when new data coming from the true data generating process is considered, after constraining the model with a given dataset.”

“Per the discussion in Section 3, this is, up to a constant and a sign flip, nothing more than the population counterpart to the test log predictive density (Gelman, Hwang, and Vehtari 2014) commonly employed in variational inference evaluation.”

“Smooth continuous models are not well suited for detecting trend discontinuities, and the model information criteria—DIC, WAIC, and LOOIC 19 —shown in Table 1 14 differed only slightly among these five models, thereby indicating that no one model performed significantly better than any of the others.”

“Following Gelman et al. (2014), the WAIC for model A is given by: Here, (cid:99) lpd A denotes an estimate of the log point-wise predictive density: A is a generic notation that refers to the s th draw of the parameters and latent states from the full posterior distribution of model A .”

“WAIC is a Bayesian analog to AIC and approximates the predictive accuracy of leave-one-out crossvalidation (Gelman et al., 2014).”

“This is a very similar thing to the expected predictive log density (ELPD) [54], with one major difference being that the latter requires that the true data generating process be known (or can be sampled from).”

“The WAIC can be seen as an approximation to the cross-validation criterion (Gelman et al. 2014), computed as WAIC = − ∈ lppd + 2 p WAIC , where lppd is the log pointwise predictive density, with lppd = ∑ = 1 log ∫ p ( y i � ) p ( � y ) d , where 𝝑 = ( T , b T , d T ) T and the penalization i 𝝑 )…”

“In Gelman et al. (2013) they compare various model selection methods including AIC and cross-validation.”

“Model comparison was based on mean absolute error (MAE) and Watanabe–Akaike information criterion (WAIC), an information criterion used to assess the predictive accuracy of Bayesian models [33].”

“According to a predictive model selection criterion (leave-one-out cross-validation [20]), the Additive-CHMM had superior accuracy compared with a set of site-specific standard HMMs and other formulations that we developed (see electronic supplementary material, table S1).”

“Careful judgement should be applied to this task, although model comparison methods can also be helpful [6, 13, 14].”

“1 permits to reduce the parameter search space in a convenient way for choosing the priors in ranges between 0 and approximately 1, which brings three main advantages: (i) independent prior distributions for each parameter can take the same form, (ii) by introducing scale information of the original units in which the…”

“WAIC is superior to DIC under various settings; for example, if the posterior deviates from multivariate normality [8].”

“…the appropriateness of our proposed model for the observed data by computing posterior predictive p-values for five different test quantities including the mean, standard deviation, maximum, proportion of observations that equal zero, and the chi-square discrepancy function (Gelman et al., 2014).”

“…for a training and validation set (corresponding to 70% and 30% of randomly selected observations), the Watanabe–Akaike information criterion (WAIC, Gelman, Hwang, and Vehtari 2014) and by predictive analyses supported by the cross-validated probability integral transformation (PIT)…”

“The goal of using elpd_loo is to select models with the best predictive performance and generalizability, providing a robust estimate of out-of-sample accuracy, which accounts for both model complexity and parameter uncertainty within the Bayesian framework [56, 57].”

“We implemented four model comparison methods that rely on a measure of out-of-sample prediction error [23,24].”

“…a model selection procedure for each of our questions using leave-one-out cross validation information criterion scores (LOOIC) [63] to determine the most parsimonious model for each question (Table 3); that is, the simplest model within two LOOIC scores of the best-fitting model [63, 64].”

“Candidate models were compared using the widely applicable information criterion or Watanabe-Akaike information criterion (WAIC) whereby the models with lowest WAIC values were considered to have the best fit [34, 35].”

“We can use draws from the posterior predictive distribution of α ∗ k , k ′ for our fitted model to form a posterior predictive p -value [26]: If the model fits well, the value of P (cid:31) α ∗ k , k ′ > α k , k ′ (cid:30) is expected to be close to 0.5 , with extreme values close to 0 or 1…”

“We finish the analysis of the predictive performance with a comparison of the predictive result for the multi-step forecasts for the last 22 observations in the sample without the use of rolling samples, comparing the predictive densities using the Log-Predictive Likelihood (Gelman et al. 2014), again comparing the fa…”

“TMB uses a No-U-Turn Sampling (NUTS) algorithm to estimate the growth parameters, which is an advanced Markov chain Monte Carlo (MCMC) method (Hoffman and Gelman, 2014).”

“Additionally, it includes several information criteria–namely, the widely applicable information criterion (WAIC; Gelman et al. 2014), the log pseudomarginal likelihood (LPML; Geisser and Eddy 1979), and the deviance information criterion (DIC; Celeux et al. 2006)–to facilitate model and variable…”

“It is important to note that the prior information is relevant in estimating the parameters rather than in calculating predictive accuracy (Gelman et al 2014).”

“Nine integrated likelihood estimators, described inOnline Resource 1, Section S1, are considered: the un-normalized weight estimator denoted by UWE (Bugallo et al. 2015), the estimator based on an information criterion (McElreath 2018) of which we consider six different specifications denoted by WAIC1, WAIC2, BIC, DIC…”

“Along with R2, which provides a general idea of how well a model fits the data (larger indicates a better fit), AIC, which measures the predictive accuracy of the model (smaller indicates higher accuracy) [41, 42], has been used for model selection studies involving human movement [26, 43].”

“However, the BIC is typically inappropriate for hierarchical models since the hierarchical structure implies that the e(cid:27)ective number of parameters will typically be lower than the actual number of parameters in the likelihood (Gelman et al., 2014).”

“Moreover, AIC is being supplanted by contemporary alternatives, such as WAIC (Gelman et al., 2014; Watanabe & Opper, 2010) and approximate leaveone-out cross-validation (Vehtari et al.”

“We used Watanabe- Akaike Information Criterion (Gelman et al., 2014; Watanabe, 2010) and LOO (leave- one- out) cross- validation (Vehtari et al., 2017) as additional criteria for selecting the best fit model.”

“Commonly used information criteria include AIC (Akaike Information Criterion), WAIC (Widely Applicable Information Criterion) and BIC (Bayesian information criterion) [14], all of which penalize the complexity of a model.”

“In general, a model with smaller WAIC value was better supported by data [47].”

“The Stan framework utilizes the No-U-Turn Sampler (NUTS) algorithm, an advanced variant of HMC, to efficiently sample from the posterior distribution (Hoffman and Gelman, 2014).”

“An increase in ELPD indicates improved model fit and generalisation capacity, while a decrease in LOOIC—due to its negative scaling—reflects better model performance (Gelman et al. 2014).”

“This criterion is preferred over the commonly used deviance information criterion (DIC) for model selection (37, 38).”

“Innovative predictive schemes, which leverage the projec-tivity properties and analytic results for the proposed pExP prior, are derived in Section 3.2.”

“The LPDS gives a measure of how well the model fits the out-of-sample data (Gelman et al., 2014), with larger values being preferred.”

“Nevertheless, in the face of previous results showing that DIC may be problematic in some cases (Gelman et al. 2014), we recommend the careful utilization of DIC alongside with other performance metrics with practical significance to do model selection.”

“We suggest using the Widely-Applicable Information Criteria (WAIC), which approximates leave-one-out cross-validation (Watanabe and Opper 2010; Gelman, Hwang, and Vehtari 2014).”

“We use different cross-validation and information criteria (Gelman et al 2014) to compare the prediction capacity and performance among model configurations.”

“As recommended by Gelman et al 22 for use with hierarchical Bayesian models where the number of parameters increases with the sample size, we used the Widely Applicable Information Criterion (WAIC) to select among models with different random effects.”

“LOO cross-validation with Pareto-smoothed importance sampling was used to select best-fitting models with highest ELPD (expected log pointwise predictive density, see [Gelman et al. 2014]).”

“We compare the models based on the Deviance Information Criteria (DIC; (Gelman et al., 2014)) which provides a measure of model fit penalized for the complexity of the model.”

“We used theWatanabe–Akaike information criterion (WAIC) Copyright © 2023 Wolters Kluwer [23] for model choice in model development, for example for picking knot frequency.”