Bayesian Versus Orthodox Statistics: Which Side Are You On?

“For a p value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect (Rouder et al., 2009).”

“For this reason, sooner or later, you are guaranteed to get a significant result if you run subjects long enough and stop when you get the p value you want (e.g., Wagenmakers, 2007).”

“…with orthodox statistics, unless pre-specified as possible by one’s stopping rule (Armitage et al., 1969); by contrast, with Bayes, one can always collect more participants until the data are sensitive enough, that is, B < 1/3 or B > 3; see e.g., Berger and Wolpert (1988), Dienes (2008, 2011).”

“…(i.e., Bayesian model averaging in regression, Clyde, 2016), the informative model comparison approach (e.g., Gu, Mulder, Decovi´c, & Hoijtink, 2014; Gu, 2016; Mulder, 2014, 2016), and a more flexible and subjective prior specification approach (e.g., Dienes, 2011, 2014, 2016; Gronau et al., 2017).”

“Dienes (2008, 2011) provided another analytical solution, with a corresponding online calculator.”

“, Edwards, Lindman, & Savage, 1963), and recently there has been a growing movement towards alternative analyses that overcome some of the shortcomings of null-hypothesis significance testing (NHST) and the associated p-values (Dienes, 2011; Gallistel, 2009; Johnson, 2013; Nuzzo, 2014).”

“The measure takes the form of an odds ratio: It is the probability of the data under one hypothesis relative to that under another (Dienes, 2011; Kass & Raftery, 1995; Lee & Wagenmakers, 2005).”

“It is important to note that, in contrast to NHST, the inference drawn from the actual data set is entirely independent from the planning stage (Berger & Wolpert, 1988; Wagenmakers et al., 2014; Dienes, 2011).”

“…analysis of posterior probabilities for H 0 versus H 1 is poised to emerge as a widely used alternative to NHST in psychological research (e.g., Dienes, 2011; Kruschke, 2011; Masson, 2011; Raftery, 1995; Wagenmakers, 2007; Wetzels et al., 2011; see also Dixon, 2003; Glover & Dixon, 2004, for…”

“scope of this paper; see Rouder, Speckman, Sun, Morey, and Iverson (2009), Wagenmakers, Lee, Lodewyckx, and Iverson (2008), or Dienes (2011) for accessible accounts.”

“Bayesian techniques that quantify the proportional evidence for or against independence or clustering (e.g., computing a Bayes factor, e.g., Dienes, 2011) can also be used for this purpose. null hypothesis that men and women were equally likely to be admitted is not tenable, with more men than…”

“For an overview of corrections for multiple comparisons in NHST, see Maxwell and Delaney (2004, chapter 5), and for a discussion of contrasting intuitions in multiple testing, see Dienes (2011).”

“Because research design and scientific reasoning make the sampling and testing intentions irrelevant, a researcher’s intuitions about data analysis match Bayesian interpretation, not frequentism (Dienes, 2011).”

“Second, many scholars prefer Bayesian statistics because they believe population parameters should be viewed as random (see, e.g., Dienes, 2011; van de Schoot et al., 2011).”

“It has been suggested that the common misinterpretations of NHST arise in part because its results are erroneously given a Bayesian interpretation, such as when the pvalue is misinterpreted as the probability that the null hypothesis is true (e.g., Cohen, 1994; Dienes, 2011; Falk & Greenbaum, 1995).”

“edu/bayesfactor and by Dienes at http://tinyurl.com/DienesBayesFactor.10 Downloadable software for Bayes factors in the R language is available at http://cran.r-project.org/web/packages/BayesFactor/ index.html, authored by Morey, Rouder, & Jamil. A package for frequentist and Bayesian analyses with pull-down menus, ca…”

“Unlike classical inferential frameworks, Bayesian inference can incorporate such prior knowledge [17].”

“The paradoxes mean that inferential outcome depends on more than the actual data obtained, and may depend on things which are in practice unknowable (the intentions and thoughts of experimenters; see Dienes, 2011 for explanation).”

“These priors should depend on how strongly each hypothesis follows from a theory, and not on how many hypotheses a researcher plans to test in the same data set (Dienes, 2011).”

“Appendix 1”

“…is considered to provide substantial evidence against the null, a BF of less than 1/3 is considered to provide substantial evidence in favor of the null, and a BF in between 1/3 and 3 indicates data insensitivity [see Dienes, 2008, 2011, for explanations of Bayes Factors and free online software].”

“In response to these concerns, Bayesian approaches have been developed which regard the experimental and null hypotheses as on an identical footing in terms of their ability to be evidentially supported or disconfirmed and which, of particular importance, allow one to evaluate evidence for the null hypothesis from a g…”

“Moreover, there is usually more than one way to solve each of these mistakes: for example, we focus on frequentist parametric statistics in our solutions, but there are often Bayesian solutions that we do not discuss (Dienes, 2011; Etz and Vandekerckhove, 2016).”

“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”

“Bayes Factor For each trial we calculated a Bayes factor, following the method suggested by Dienes, Coulton and Heather (2017). This quantified how likely the data were under the null hypothesis compared to the alternative hypothesis, which was defined to be an effect size taken from a half normal distribution with me…”

“…statistics are usually not considered in terms of thresholds, it is generally assumed that a Bayes Factor below 1/3 corresponds to substantial evidence in favour of the ‘null’ hypothesis, whereas a Bayes Factor above 3 corresponds to substantial evidence in favour of the ‘effect’ hypothesis 57 .”

“The accumulation of knowledge includes both discovery and confirmation, and thus the Bayesian style of statistics may become more useful than frequentist or null hypothesis statistical testing approaches.(23,51) A Bayesian approach, by design, supports incremental model building via the use of “priors” knowledge.”

“While the method of expanding the sample size until a desirable effect is found (“preferential stopping”) would invalidate conventional “frequentists” statistical analyses, it is permissible with Bayesian analysis (Dienes, 2011).”

“In that case the test just has no result, that is, the evidence is not sufficient to support a conclusion(Dienes,2011).”

“In this context, equivalence testing 111 or Bayesian analyses 112 have been proposed as suitable approaches to directly assess evidence for the alternative hypothesis against evidence for the null hypothesis 113 .”

“…recent publications in the field of Psychology suggest using a Bayesian approach, and particularly a Bayes Factor Analysis, for the test of alternative theoretical models that may include a null-model, i.e., a model that predicts effects that are inconsequential on practical grounds (Dienes, 2011).”

“Statistical inference In this study, we used Bayes factors (Dienes, 2011; Jeffreys, 1998; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, 2007) to determine the evidence for the null and alternative hypotheses.”

“Unfortunately, all p-value inference is plagued by the same conceptual and practical problems (e.g., Dienes, 2011; Wagenmakers, 2007; Wagenmakers, Lee, Lodewyckx, & Iverson, 2008; Wagenmakers et al., in press; Wagenmakers et al., 2016; Wagenmakers, Morey, & Lee, in press).”

“…assess whether the lack of priming stemmed from insufficient sensitivity of our experimental design or from a genuine lack of effect, we computed Bayes factors (referred to as B hereafter; see Jeffreys, 1961; Dienes, 2011; 0.33 < B < 3 suggests insensitivity, and B < 0.33 implies no effect).”

“Bayesian analyses allow conclusions to be drawn about whether non-significant results provide conclusive support for the null hypothesis, relative to a pre-existing theory (Dienes, 2011).”

“Bayesian statistics are becoming more common in academic research [28].”

“…neither evidence for nor against the null hypothesis, and even when the latter is true, the p-value can take almost any value ( Cumming, 2012; Dienes, 2011).4 Thus, even when a theory lacks empirical support and when failed attempts at replication accumulate, it is still hardly possible to…”

“A natural follow-up to our third highlighted source (Dienes, 2011).”

“Alternative paradigms such as Bayesian statistics (see, e.g., Dienes, 2011; Wagenmakers, 2007) in combination with a higher threshold for the minimal amount of required evidence (e.g., Etz & Vandekerckhove, 2016) would largely remedy this problem but are still controversial because of the…”

“Notice the 95% CI includes a 6% difference, so in fact nothing follows from the non-significant result: It is as consistent with there being a relevant effect as with there being none.2 2 Non-significant results can also usefully be interpreted with Bayes Factors (Dienes, 2008, 2011).”

“75 would indicate a weak effect (32, 33)], and a Bayes factor in support of the null hypothesis over 3 indicates substantial support for the null (32).”

“The probability of our hypothesis being true given the observations (data) on these connectivity-behavior correlations was tested by a Bayesian approach (Dienes, 2008, 2011).”

“3 is seen as providing evidence in favor of the null hypothesis and a BF >3 as strong evidence against it (17).”

“One approach that offers an elegant means to do so is the computation of a Bayes factor (e.g., Dienes, 2008; Dienes, 2011; Jeffreys, 1961; Morey & Rouder, 5 Assuming an effect size d = .218, as found by Strick et al. (2011) in their meta-analysis, the power for the statistical test of this…”

“Across the six studies, all Bayes factors estimated the null hypothesis against an alternative hypothesis described by a half-normal distribution, with both the mean and the standard deviation of the difference between conditions equal to 0.5 units (see Dienes, 2011, p. 287).”

“The same process of successive strengthening of belief or factual congruency can be applied to ecological problems, and it is often far more appropriate than classical approaches (Dienes, 2011).”

“It is widely recognized in statistics (Gelman, Carlin, Stern &Rubin, 2004; Jaynes, 2003) and, increasingly, in psychology (Dienes, 2011; Kruschke, 2010, 2011; Lee & Wagenmakers, 2005) that the Bayesian approach offers a complete and coherent framework for making inferences using models and data.”

“To further verify the null results obtained in the bootstrap analyses, we conducted the Bayes factor (BF) analysis (Dienes, 2011) to assess the likelihood of data-driven alternative hypothesis over the likelihood of data-driven null hypothesis (BF10) or the reverse (BF01) with default priors for paired t-test design (…”

“When reporting nonsignificant differences, we complemented standard “frequentist” tests with Bayes factors to distinguish between an insensitive test (not providing evidence in favor or against the null hypothesis) and a genuine absence of difference (Dienes, 2011).”

“…Wicherts, & van Assen, 2017), perhaps even more so, given that Null Hypothesis Significance Testing (NHST), as regularly implemented in psychological research, is poorly suited to assess the plausibility of the null hypothesis (Dienes, 2011, 2015; Hoekstra, Finch, Kiers, & Johnson, 2016).”

“As the inherent limitations of null-hypothesis significance testing constrain the degree to which one can determine the strength of evidence in favor of the null hypothesis (34), we also conducted a Bayesian analysis that enabled us to quantify the evidence in favor of the null hypothesis H0 (RS = 0) relative to the r…”

“evidence for the null hypothesis; and a Bayes Factor in between 1/3 and 3 indicates data insensitivity (Dienes, 2011).”

“This value is above 3, which is considered to indicate substantial evidence in favor of the null hypothesis [Dienes, 2011, 2014].”

“In Bayesian inference, credible intervals represent the combination of prior beliefs and likelihood into final beliefs (Dienes, 2011).”

“This procedure was repeated until there was no difference in RT between the two task contexts, as assessed with Bayes factor (BF) analysis (Love et al., 2015) in each participant separately (BF < 1 taken as evidence for no difference between conditions; Dienes 2011).”

“To further test this claim, we calculated the Bayes factor [41, 42] comparing the null hypothesis that best-match rotations were randomly distributed to the alternative hypothesis that best-match rotations were more often consistent with chamber geometry (see Supplemental Experimental Procedures).”

“to Dienes (2011), Rouder et al.”

“In addition, Bayes Factors (alternative BF/null BF; Wagenmakers, 2007; Dienes, 2011) were estimated in JASP v0.”

“Critics of the traditional statistical inference method of significance testing argue that “it is time for researchers to consider foundational issues in inference” [10, p.274].”

“We performed Bayes tests for all results yielding non-significant differences (Dienes, 2011, 2014; Dienes, Coulton, & Heather, 2017; Dienes & Mclatchie, 2017).”

“A BF quantifies the relative probability of the data under, e.g., the null versus the alternative hypothesis (Dienes, 2011; Gallistel, 2009; Morey, 2015; Rouder, Speckman, Sun, Morey, & Iverson, 2009).”

“Group-level statisticsAll statistical analysis at the group level was based on Bayesian statistics (Dienes, 2011; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wetzels et al., 2011) using the R library BayesFactor (Morey & Rouder, 2014).”

“…govern our sample size, knowing that the Bayes factor becomes progressively smaller with increasing sample size in cases where the null hypothesis becomes more and more likely than one’s alternative hypothesis (Dienes, 2011; Rouder, 2014; Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2017).”