Bayesian Versus Orthodox Statistics: Which Side Are You On?

“For a p value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect (Rouder et al., 2009).”

“For this reason, sooner or later, you are guaranteed to get a significant result if you run subjects long enough and stop when you get the p value you want (e.g., Wagenmakers, 2007).”

“We used Dienes’ (2011) calculator.”

“First, to interpret a null effect as evidence for there being no effect (in this case no localisation), would need alternate statistics, such as a Bayes Factor (Dienes, 2011), to assess the degree of evidence for ‘no effect’.”

“…hypothesis signi fi cance testing and the associated p -value has many shortcomings, for example it relies on hypothetical data and can be easily manipulated — with larger sample sizes able to make small differences signi fi cant without any practical value ( Jarosz & Wiley, 2014 ; Dienes, 2011 ).”

“A Bayes factor was calculated, with the predictions of H1 modeled as a half-normal with an s.d. of 124 ms (as per Dienes, 2011, forthcoming), giving B¼ 0.28.”

“Consequently, it is not susceptible to the forking paths problem (Dienes, 2011; see also Bender & Lange, 2001).”

“E-mail: lincoln@colling.net.nz newed interest in the conceptual foundations of significance testing and renewed criticism of the procedures themselves (e.g., Wagenmakers, 2007; Dienes, 2011; Szűcs and Ioannidis, 2017b).”

“…knowledge (all 1.12 lV or greater), and also excludes the size of P3 effect found for explicit triplets reported above (i.e., .73 lV).4 Bayes Factors can be used to establish the strength of evidence for a theory predicting an effect over the null hypothesis, or vice versa (Dienes, 2008b, 2011).”

“However, our assessment of whether the synesthetic Congruency effect supports the prediction of the dual-coding hypothesis using the Bayes factor (Dienes, 2011) was restricted to synesthetes and thus is not hindered by this confound.”

“Bayes factors were also calculated to interpret null findings and assess the strength of evidence [63–65].”

“Under the Bayesian framework, it is possible to combine data sets and as one adds more data one is simply adding more evidence to prior (however vague) knowledge (Dienes, 2011).”

“Bayes factors below 0.3 and above 3 are evidence for the null and the alternative hypothesis, respectively, while values in between indicate that the data are insensitive (Dienes, 2011).”

“The DSQ is a widely used self-report measure of empirically derived groupings of defense mechanism ranked on an adaptive hierarchy.”

“Values in between indicate data insensitivity (see [32,55], for explanation of Bayes Factors and free online software).”

“It cannot be negative [35].”

“Whereas standard hypothesis testing assumes that the null hypothesis is correct and evaluates whether it can be rejected based on the evidence, Bayes factors measure the relative strength of evidence in favor of both the alternate and the null hypothesis (Dienes, 2011; Wagenmakers, 2007).”

“33 and 3 indicates merely anecdotal or inconclusive evidence for either hypothesis [68,69].”

“That is, the null result could be due to a test that is not sensitive enough to pick up a population difference between the two groups being compared (Dienes, 2011).”

“This model now appears more and more frequently within the field of empirical psychology (see Dienes, 2011).”

“Additionally, Bayesian methods seem to be much more closely aligned with common human intuitions and ways of reasoning about the interpretation of statistical tests in general and the notion of significance in particular [35, 37, 38].”

“First, it appropriately drew attention away from statistical “signi cance levels,” which are fraught with interpretive problems (Rozeboom, 1960; Dienes, 2011), and towards more informative measures of effect size (Wilkinson & The Task Force on Statistical Inference, 1999).”

“If one were to take this approach, a “half-normal” or “half-Cauchy” onetailed distribution would be a suitable form for the alternate prior distribution (Dienes, 2011; Rouder & Morey, 2011).”

“…results Prior to scanning, the stimulus set was titrated to match reaction times between the two tasks for each participant separately (assessed with Bayes factor analysis for each participant separately, using a threshold of BF < 1, all actual BF10 < .76) (Dienes, 2011; Love et al., 2015).”

“Bayes factors 0.33 provide substantial evidence for the null over the alternative hypothesis, whereas Bayes factors 3 can be interpreted as evidence for the alternative over the null hypothesis (Dienes, 2011).”

“Bayes factors were computed for critical absence of effects observed, to distinguish between the lack of sensitivity of tests and genuine absence of difference (Dienes, 2011).”

“Bayes factors (BF) computed in JASP (2017) are reported for main results, allowing interpretations in favor of the H0 (no group differences): BF < 1/3 can be interpreted as evidence for H0, BF > 1/3 and < 3 as no evidence to speak of, and BF > 3 is evidence for H1 (see Dienes 2011).”

“…alternatives to traditional ways of conducting statistical testing are not infrequently various forms of Bayesian hypothesis testing (see, e.g., Dienes, 2011; Kruschke, 2014; Lee & Wagenmakers, 2013; Rouder et al., 2009; van Ravenzwaaij et al., 2019; van Ravenzwaaij & Wagenmakers, 2021; see…”

“…a Bayesian approach to hypothesis testing, and a number of researchers have recently argued for a more widespread adoption of this approach (e.g., Dienes, 2011; Lee & Wagenmakers, 2005; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, 2007; Wagenmakers, Lodewyckx, Kuriyal, & Grasman,…”

“is considered weak evidence [45].”

“The use of prior knowledge is commonly described as the prior distribution that can be represent previous findings in the literature or in cases when there is less previous knowledge to inform the model, the prior distribution is more uncertain or wider (Dienes, 2011).”

“NHST does not provide clear rules for stopping data collection; therefore, as long as sample size increases any H0 can be rejected (see [9, 18]).”

“This factor is substantially less than 1, providing some support for the null (for Bayes factor calculator, see Dienes, 2011); however, it does not establish it (which it would if less than 0.1).”

“Using the same criteria employed to calculate the Bayes Factor in Experiment 2A, the non-significant [t(35) = 1.07, p = 0.294, r = 0.18] difference between the two conditions returned a Bayes Factor of 0.58 a value that cannot be taken as evidence for nor against the theory under test (Dienes, 2011) and is therefore n…”

“For instance, the prior for a mean or a regression slope is often modeled by the normal distribution. m, s2 Its parameters are m, the mean, a location parameter that determines its center, and s2, the variance, a scale parameter that determines its spread.”

“As Fisher said, after a negative study, one can only “withhold judgment” (Dienes, 2011); but negative studies are harder to publish and one can’t help but wonder if publication bias is at work here, especially for the smaller, uncontrolled studies (Dwan et al., 2008).”

“While the tests were non-significant in the latter two cases, it is not clear whether this counts as evidence against the application of UTT to conscious knowledge or whether the evidence is just insensitive (Dienes, 2011).”

“The common misinterpretation of a confidence interval is frequently attributed to viewers erroneously interpreting the range from a Bayesian perspective [12, 16].”

“Following the guide of Dienes (2011), we performed a Bayesian model comparison based on these parameters and calculated a Bayes factor of 0.26.”

“Of note, the use of a Bayesian inferential framework does not require adjustment for multiple comparisons as is practice in frequentist inference (Dienes, 2011).”

“Bayes Factors are presented within the brackets.”

“A number of recent papers describe advantages of adopting Bayesian inference methods, both in the social sciences [2,8, 24,35,36] and in HCI [31,42].”

“The Bayes Factor was .97 indicating the evidence was insensitive in distinguishing the hypothesis of a quadratic from the null hypothesis (.33 and below being the cut off for strong evidence for the null, Dienes (2011)).”

“A value closer to 1 suggests that the result is insensitive and any difference is Banecdotal^ (Dienes 2008, 2011, 2014, 2016).”

“…resting state sessions. tDCS (sham or real, in counterbalanced order) was applied for 25 min starting from the beginning of the experiment. substantial evidence for the null hypothesis (H0), and values between 0.333 and 3 are deemed inconclusive and indicate a lack of sensitivity (Dienes, 2011).”

“Thus, the process of adding participants, and the consistent usage of Bayes factors throughout (see below), ensured our study was not biassed towards favourable results (Dienes, 2011).”

“We rejected the null hypothesis for the values of p close to 0.05 when the Bayesian analyses pointed out evidence in favor of the alternative hypothesis (Dienes, 2011), namely, when the posterior probability was equal or bigger than 0.95.”

“, that species X does not show conditioned place avoidance) relative to the alternative, whereas standard frequentist statistics cannot provide evidence for the null hypothesis (Dienes, 2011).”

“Since the orthodox (Neyman and Pearson) statistical approach is unable to distinguish between these two possibilities, we adopted a Bayesian approach (Dienes, 2011).”

“One statistical approach well suited for addressing this limitation of NHST in providing support for the null hypothesis is Bayesian analysis (Dienes, 2011).”

“The results conceptually replicate those of Dienes and Hutton (2013) who showed that applying rTMS to the left DLPFC increased subjective ratings of hypnotic experience, compared to stimulation of the vertex.”

“For more detailed information about Bayesian inference, see, for instance, O’Hagan and Forster (2004), Kruschke and John (2010), Lee and Wagenmakers (2010), and Dienes (2011).”

“Although for simple models the value of Bayes factor can be derived analytically as shown by [14]–[17], for complex models it can only be computed numerically using for example the Savage-Dickey (SD) method [18]–[20].”

“General data analyses Data were analyzed with both frequentist and Bayesian statistics (Dienes, 2011), except for voxel-wise brain analyses that were only analyzed using frequentist approaches.”

“Using these assumptions, we employed the Bayes factor calculator provided by Dienes (2011) with a half-normal distribution, and this gave a Bayes factor of 3.18 for the contrast between upright familiar and inverted familiar category exemplars, confirming that we can be confident of this inversion…”

“The Bayes Factor is a number between 0 and infinity, where values greater than three indicate strong evidence for the alternative hypothesis (H1), values less than 1/3 indicate strong evidence for the null (H0), and values between 1/3 and 3 indicate lack of sensitivity [56].”

“In contrast to tNHST, Bayesian change point analysis provides a tool for evaluating the strength of the evidence in favor of various models of change in the probability of obtaining a value in a chronological record (Dienes, 2011; Lee & Wagenmakers, 2005; Salsburg, 2001).”

“Unplanned further analyses were undertaken to examine whether non-significant associations could best be characterised as evidence of no effect or whether data were insensitive (Depaoli, Rus, Clifton, van de Schoot, & Tiemensma, 2017; Dienes, 2011, 2016).”

“We chose to do this in a Bayesian framework, partially because Bayesian analyses are robust to multiple comparisons (Dienes, 2011; Gelman et al., 2012, 2013).”

“Thus, underlying subjective confidence is the implicit belief (common among researchers but challenged by statisticians, see Dienes, 2011; Schervish, 1996) that the statistical level of significance is diagnostic of the correctness of the tested hypothesis, as well as the likely reproducibility of…”

“…indicates weak support for the alternative hypothesis (note: Jeffreys (1998) interprets a BF 1–3 as anecdotal evidence for the alternative hypothesis; Dienes (2011) suggests this indicates experimental insensitivity), whereas a BF > 3 suggests strong evidence for the alternative and a BF > 10…”

“Confidence intervals for all estimated correlations include 0 but are wide, hence we report also Bayes factors (BFs)6, which can help determine whether these non-significant results support the null hypothesis, or whether the data are rather just insensitive (Dienes, 2011).”

“Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side”

“The idea that coherence can serve as a cue for correspondence underlies the oft-criticized interpretation of statistical level of confidence (Schervish, 1996; see Dienes, 2011).”

“Bayesian model comparison, on the other hand, can find either the null or alternative hypothesis more probable [19,20].”

“It was assumed that smaller differences in slope were more likely than large differences in this kind of data, so the prior hypothesis tested a half-normal distribution with a mean of zero and standard deviation equal to the difference from the previous study (Dienes, 2011).”

“The Bayes factor gives the ratio with which the alternative hypothesis is favored over the null hypothesis (i.e., larger BFs argue in favor of the alternative hypothesis; see Dienes, 2011, for an overview).”