Bayesian Versus Orthodox Statistics: Which Side Are You On?

“For a p value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect (Rouder et al., 2009).”

“For this reason, sooner or later, you are guaranteed to get a significant result if you run subjects long enough and stop when you get the p value you want (e.g., Wagenmakers, 2007).”

“Comparisons were made using both traditional null-hypothesis significance testing (NHST) and Bayesian analysis, as the latter allows for a more robust examination of the null hypothesis (Dienes, 2011).”

“For statistical testing, we used Bayes factors as implemented in the R package BayesFactor (Morey & Rouder, 2015), a Bayesian equivalent of paired t tests (Dienes, 2011; Rouder et al., 2009).”

“Following convention, we interpreted BFs greater than 3 as providing moderate (or greater) evidence for the alternative hypothesis, less than 0.33 as moderate evidence for the null hypothesis, and between 0.33 and 3 as insensitive evidence (Dienes, 2011; Jeffreys, 1961).”

“…testing is the Bayes factor, which quantifies the relative plausibility of the data under two competing hypotheses (for overviews, see e.g. Dienes, 2011, 2014, 2016; Kruschke & Liddell, 2017; Jarosz & Wiley, 2014; Morey & Rouder, 2011; Rouder & Morey, 2012; Rouder, Speckman, Sun, Morey, &…”

“They take account of previous knowledge, which distinguishes them from frequentist statistics (Dienes 2011).”

“Within this statistical framework, the data models are defined explicitly using intuitive notions and knowledge about the relationships between different features and their distributions (Dienes, 2011; Kruschke, 2011b) via expert elicitation, knowledge, and experimental findings to inform priors for Bayesian statistic…”

“Regarding the sensitivity criterion, our Bayesian approach made it possible to convincingly conclude [BF > 30 (Dienes, 2011)] that our null findings can be interpreted as offering support for the absence of differences rather than as resulting from a lack of sensitivity (Vadillo et al.”

“Furthermore, bayesian analysis has the advantage of mitigating the multiple comparisons problem because there is no need to correct for multiple testing (Dienes, 2011).”

“…with which the alternative hypothesis is favored over the null hypothesis: larger BF10 values argue in favor of the alternative hypothesis, with values above 3 denoting “substantial evidence” in favor of the alternative hypothesis; values less than 1 favor the null hypothesis; see Dienes (2011).”

“…that the perceptually equalized phase would produce a mean illusion effect onMGA that was 422% of the mean illusion effect on MGA found during illusion-adaptation ****Half of model-predicted mean was used as standard deviation based on a suggestion by Dienes (2011) keypad held in the left hand.”

“BFs below 0.33 and above 3 were interpreted as moderate evidence for the null and alternative hypotheses respectively, whereas BFs between these values were interpreted as inconclusive evidence (Dienes, 2011; Jeffreys, 1961).”

“Using the procedure outlined by Dienes (2011), we first conducted a Bayes analysis on the Face Orientation by Stimulation interaction found in both Experiment 1 and 2.”

“Evaluators can introduce controls for multiplicity of contrasts, but doing so is controversial (Dienes, 2011).”

“The zero-correlation criterion (Dienes & Berry, 1997) posits that knowledge is unconscious when there is a lack of correlation between confidence and accuracy. The guessing criterion (Cheesman & Merikle, 1984) states that if people perform above chance when they believe they are guessing, the knowledge is unconscious.…”

“With this assumption the Bayes Factor (using the online calculator for the website for Dienes 2008) was 0.85.”

“Frequentists statistics concerns the probability of data given a theory, whereas Bayesian statistics concerns the probability of a theory given the data [Dienes, 2011].”

“Contrary to NHST, Bayes factors allow such conclusions on whether the evidence supports H0, the evidence supports H1, or the evidence is ambiguous (Rouder et al., 2009; Dienes, 2011, 2014; Lee and Wagenmakers, 2013).”

“…recommended by many authors, as it is able to overcome the often-mentioned limitations of the frequentist approach (e.g., Type 1 and 2 errors) and can give a more direct answer to research questions than the classic (aka frequentist) analysis (Dienes, 2011; Kline, 2013; Jarosz and Wiley, 2014).”

“Statisticians and methodologists of Bayesian conviction have often argued against the use of p-values and classical null-hypothesis significance testing (NHST) in recent years (w07; e.g., Dienes, 2011; Krueger, 2001).”

“In experimental psychology, demonstrations of the benefit of informed/specific Bayes factor hypothesis tests includes Vohs et al. (in press), Gronau et al. (2017), Ly et al. (2019) and in particular the work of Zoltan Dienes (e.g., Dienes, 2008; 2011; 2014; 2016; 2019).”

“To determine whether there was evidence for no effect of the suggestion, we used a Bayes factor (Dienes, 2008, 2011), where we contrasted the theory that the suggestion had some effect with the null hypothesis that the suggestion had no effect.”

“It is important, however, to determine whether this non-significant result supports the null hypothesis (that there was no difference in time spent in each zone), or supports no conclusion at all (Dienes, 2011).”

“In contrast to LMEs, the Bayesian approach is immune to multiple comparison problems (Dienes, 2011).”

“This ratio compares and quantifies the evidence in favor of or against the null model (Dienes, 2011; Kruschke, 2011).”

“In our analyses, we solely employed estimation procedures instead of testing the existence of differences with an inferential statistical tool such as the null-hypothesis significance test (Fisher, 1925; Neyman & Pearson, 1933) or the Bayes factor (e.g. Dienes, 2011; Rouder et al., 2009).”

“We used a Bayes Factor (BF) hypothesis-testing approach (Dienes, 2011) and conducted a Sequential Bayes Factor design, as proposed by Schonbrodt, Wagenmakers, Zehetleitner, and Perugini (2015).”

“A Bayes factor is useful for indicating continuous degrees of support for a hypothesis and hence when a null result counts against a theory that predicts a difference or doesn’t count one way or the other (see [49,50]).”

“Additionally, in Experiment 3 we computed the Bayes factors (Dienes, 2011) to investigate further the level of confidence for the effects found.”

“In particular, some researchers have advocated the use of Bayes factors (e.g., Dienes, 2011; Rouder et al., 2009; Wagenmakers, Lodewyckx, Kuriyal, & Grasman, 2010; see Kruschke, 2011, for an alternative Bayesian approach).”

“Thus, we 1 The logical correctness of Bayesian inference aside (Dienes, 2011), much of reality seems to be robust to the use of “irrational” frequentist tools, akin to the “robust beauty of improper linear models” (Dawes, 1979).”

“To interpret the latter non-significant results, Bayes factors were used (Dienes, 2011).”

“Even though the use of NHST is common in user research and elsewhere, several limitations of have been noted in the behavioural-research literature, with cogent calls for abandoning NHST altogether (e.g., Cohen, 1994; Cumming, 2014; Dienes, 2011; Rozeboom, 1997).”

“We also performed a Bayesian Repeated Measures ANOVA (Rouder et al., 2012, 2016) in the software program JASP (Love et al., 2015; Wagenmakers et al., 2016) with a default uniform distribution of prior model probabilities (for arguments for the Bayesian approach, e.g., Dienes, 2011).”

“…of the data supporting an absence of interaction (H0) is computed by dividing the BF of the interaction model against the BF of the full model without interaction, resulting in a BF01, where BF01 > 3 (i.e., H0 three times more likely than H1) is set as the significant threshold (Dienes 2011).”

“To complement the frequentist t-test comparison, a BF was calculated with the Dienes BF calculator (Dienes, 2011, 2014), with the prior distribution defined as a half-normal distribution with a maximum probability at 0 ms and standard deviation of 23 ms.”

“0 as providing substantial evidence in favor of the null and alternative hypotheses, respectively, and in-between values as reflecting insensitive evidence (40,41).”

“Following the recommendations by Dienes (56), we modeled the prediction for the hypothesis that there is temporal binding as a two-tailed normal distribution with a mean of 33 ms and a SD of half the mean.”

“We found BF 0.29 ( 0.33), which strongly supports the null (i.e., no difference) over the alternative hypothesis (Dienes, 2011).”

“Bayes factors between one third and three suggest that the data are insensitive to distinguish the theories (Dienes, 2011).”

“…ratio 456 between the probability of the alternative hypothesis, given the observed data, and the probability 457 of the null hypothesis, given the observed data, then a BF can tell us how probable the null 458 hypothesis is relative to the alternative (and vice versa; e.g., Dienes, 2011; 2014).”

“31), which supports our acceptance of the null hypothesis (Dienes, 2011).”

“In recent years, this weakness has led to an interest in the use of Bayesian hypothesis testing as an alternative method to assess the validity of the null hypothesis [75,76].”

“In each case, the probability of the alternative hypothesis is much smaller than what is typically considered even a weak effect: A value of pBIC(H1jD) between 0.5 and 0.75 is considered a weak effect (Raftery, 1999; Dienes, 2011).”

“Relative to frequentist statistics, the Bayesian approach has the advantage of relying on a model comparison rationale including the null model, thus allowing to quantify the strength of evidence associated with as well as against each model (Dienes, 2011; Morey & Rouder, 2011; Wagenmakers, 2007).”

“Because the outcome of Bayesian parameter estimation does not depend on sampling intentions or stopping rules (Dienes, 2011), we made use of the opportunity to further increase precision of parameter estimation and included these participants in the experiments and analyses.”

“Ce modèle est de plus en plus utilisé dans le champ de la psychologie empirique [12].”

“Since an extended discussion of Bayesian inference is beyond the scope of this article, we point interested readers to the work by Dienes (2011), Hoijtink (2013), and Rouder et al. (2009).”

“Given that it is about relative evidence, a BF of about 1 does not bear any evidence in favor of either one of the hypotheses (e.g., Dienes, 2011; Gallistel, 2009; Rouder, Speckman, Sun, Morey, & Iverson, 2009).”

“For this reason, with Bayes factors, one does not have to worry about corrections for multiple testings (see Dienes, 2011).”

“These statistics are also not affected by the multiple comparison problem (Dienes 2011).”

“An alternative to p values is to use Bayes factors (e.g., Dienes, 2011; Kass & Raftery, 1995; Kruschke, 2013; Lee & Wagenmakers, 2005; Rouder, Speckman, Sun, Morey, & Iverson, 2009).”

“However, Bayesian statistics are affected less by this multiple comparison problem (Dienes, 2011), and still point towards substantial evidence for an association between arithmetic and the volume of the right fusiform gyrus.”

“Second, SAs were conducted to examine whether observed nonsignificant associations for the primary analysis could best be characterized as evidence of no effect or whether the data were insensitive to distinguish the null from the alternative hypothesis [18].”

“the reliability of hypothesis tests in experimental psychology (Dienes, 2011; 2014; Wagenmakers et al., 2010).”

“To further extend past research, this study used a Bayesian analytic approach, which relies more upon model fit indices and effect sizes in determining model adequacy, rather than arbitrary significance tests and untenable assumptions (Dienes, 2011; Wagenmakers, 2007).”

“Since a non-significant t-test cannot provide unequivocal evidence in favour of the null hypothesis, we additionally performed a Bayesian analysis (Dienes, 2011; Sterzer et al., 2014) using the statistical software JASP 0.7.5.5.”