On the logic and purpose of significance testing.

“In the psychological literature, the pros and cons of NHST have been, and continue to be, hotly debated (see, e.g., Cohen, 1994; Cortina & Dunlap, 1997; Cum-ming, 2007; Dixon, 2003; Frick, 1996; Gigerenzer, 1993, 1998; Hagen, 1997; Killeen, 2005a, 2006; M. D. Lee & Wagenmakers, 2005; Loftus, 1996,…”

“Researchers rely heavily on p-values to decide about the importance of the evidence obtained in experimental and non-experimental designs (Cortina & Dunlap, 1997).”

“…The Netherlands e-mail: r.hoekstra@rug.nl J. N. Rouder University of Missouri, Columbia, MO, USA defended its usability (e.g., Abelson, 1997; Chow, 1998; Cortina & Dunlap, 1997; Winch & Campbell, 1969), there seems to be widespread agreement that the results from NHST are often misinterpreted.”

“There is nothing wrong with hypothesis testing and p-values per se as long as authors, reviewers, and action editors use them correctly (Abelson, 1997; Bettis, et al., 2016; Cortina & Dunlap, 1997; García-Pérez, 2016; Murtaugh, 2014).”

“Furthermore, blindly interpreting confidence intervals only against the standard of whether zero point is included is nothing more than rejecting or not rejecting on the basis of p-value (Cortina & Dunlap, 1997; Thompson, 1999a, 2008).”

“Schmidt (1996) showed that the possibility of a type II error easily exceeds .40 (see also, Hunter, 1997; for discussions, see Abelson, 1997; Cortina & Dunlap, 1997; Estes, 1997).”

“…H 1 as with H 0 and is grounds only for indecision or suspension of judgment with respect to the truth of H 0 (Fisher 1925, Tukey 1960, 1991, Kalbfleisch & Sprott 1976, Oakes 1986: 31, Abelson 1995, Cortina & Dunlap 1997, Harris 1997a, 1997b, Nickerson 2000, Tryon 2001, Lombardi & Hurlbert 2009).”

“Although this approach provides more information than null hypothesis significance tests, studies that report confidence intervals often focus on whether they exclude zero, which is logically equivalent to null hypothesis significance testing itself (Cortina and Dunlap, 1997).”

“To illustrate this, even those who see value in statistical significance testing (e.g., Abelson, 1997; Chow, 1996, 1998; Cortina & Dunlap, 1997; Hagen, 1997; and Mulaik, Raju, & Harshman, 1997) simply never bring up, much less defend, the issue of the adequacy of the p value as a measure of…”

“…is to treat the null hypothesis not as a single value but as a range of values that can be considered negligible from a theoretical standpoint (Binder, 1963; Cortina & Dunlap, 1997; Fowler, 1985; Hodges & Lehmann, 1954; Meehl, 1967; Murphy, 1990; Serlin, 1993; Serlin & Lapsley, 1985; Tryon, 2001).”

“However, the confidence intervals were not exactly precise for accepting the null hypothesis (Cortina & Dunlap, 1997; Frick, 1996).”

“The trade-off between precision and accuracy and the question of which to prioritize echoes an ongoing discussion in the field about alternative goals in conducting research (e.g., Abelson, 1985; Cohen, 1990, 1994; Cortina & Dunlap, 1997; Prentice & Miller, 1992; Schmidt, 1996).”

“However, the Shapiro-Wilk test is highly sensitive to large sample sizes and may reject the null hypotheses of normal distribution even if the departures from normality are trivial (Cortina & Dunlap, 1997).”

“Schmidt (1996) shows that the possibility of a Type II error easily exceeds .40 and, therefore, strongly advises against the use of statistical significance in interpreting research findings (see also, Hunter, 1997; for discussions, see Abelson, 1997; Cortina & Dunlap, 1997; Estes, 1997).”

“Given that the information contained in a hypothesis test is comparable to that in a confidence interval (cf. Cortina & Dunlap, 1997), we choose the latter, computing the 95% confidence interval as: 4 ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ± n Q ) 96 .”

“The probabilistic nature of these statements can cause problems, but this is not especially likely (Cortina & Dunlap, 1997).”

“The fact that the typical reasoning from Fisher’s disjunction is logically invalid is well-known (e.g., Beck–Bornholdt & Dubben, 1996; Cohen, 1994; Cortina & Dunlap, 1997; Falk & Greenbaum, 1995; Falk, 1998; Pollard & Richardson, 1987; Krämer & Gigerenzer, 2005; Rouder, Morey, Verhagen, Province,…”

“For Fisher, a good control of the research design (Cortina and Dunlap, 1997; Fisher, 1955; Johnstone, 1987), especially random allocation, is paramount to make sensible inferences based on the results of tests of significance (Fisher, 1954; Neyman, 1967).”

“…some researchers have argued that p values can provide an interesting answer to a statistical question whenever researchers want to make an ordinal claim about the direction of an effect (Abelson, 1997b; Chow, 1988; Cortina & Dunlap, 1997; Hagen, 1997; Haig, 2017; Miller, 2017; Nickerson, 2000).”

“Furthermore, researchers who mindlessly interpret confidence intervals only against the standard of whether the interval subsumes zero are doing nothing more than a mindless ‘nil’ hypothesis test (Cortina & Dunlap, 1997).”

“An example of missing the bigger contribution is provided by my own study of the effects of trade on market orientation (Ellis, 2007). In Table 4 of that study I compared the market orientation of two groups of firms. As predicted, the 74 firms that were highly dependent on distant markets were found to have lower mar…”

“…has its defenders (e.g., Mulaik, Raju & Harshman, 1997), it has for decades been heavily criticized (e.g., Bakan, 1966; Batanero, 2000; Cohen, 1994; Cortina & Dunlap, 1997; Falk & Greenbaum, 1995; Harlow, Mulaik & Steiger, 1997; Lecoutre, Poitevineau & Lecoutre, 2003; Rozeboom, 1960; Schmidt,…”

“Many of the papers that I have managed to get published, including those that have been well-cited, had to overcome serious reviewer misgivings about contribution (e.g.,Cortina, 1993a, 1993b; Cortina & Deshon, 1998; Cortina & Dunlap, 1997; Cortina & Folger, 1998).”

“For reviews of arguments in favor of this approach, see Chow (1996), Cortina and Dunlap (1997) , and Hagen (1997).”

“Although there are thoughtful supporters of null hypothesis testing (e.g., Abelson, 1997; Cortina & Dunlap, 1997; Frick, 1996; Robinson & Wainer, in press), researchers increasingly recognize the limitations of the statistical method Rozeboom (1997) described as “the most bone-headed misguided…”

“Second, because confidence intervals can be interpreted as equivalent to NHST (Bedeian, Sturman, & Streiner, 2009; Cortina & Dunlap, 1997), it is unclear how NHST passes Abelson’s (1997) cognitive hurdle, but confidence intervals do not.”

“Interestingly, relatively few have defended statistical significance tests and those who have, agree with many of the criticisms leveled against such tests (e.g., Abelson, 1997; Cortina & Dunlap, 1997; Chow, 1998; Wainer, 1999).”

“De esta manera, el TE ha sido entendido como un complemento necesario para el análisis de los datos empíricos en la prueba de hipótesis tradicional (Cortina & Dunlap, 1997).”

“Frick (1996) and Cortina and Dunlap (1997) have been among the more reflective defenders of contemporary statistical testing practices (see Thompson, 1998).”

“However, in many cases, researchers aim to decide whether significant results are consistent or inconsistent with directional (dividing) hypotheses (e.g., Cortina & Dunlap, 1997, p. 168; Cox, 1977, p. 51-52; Tukey, 1991, p. 100).”

“Still others (e.g., Cortina & Dunlap, 1997; Frick, 1996) have argued that there is little or nothing wrong with contemporary analytic practices, though some of these arguments have themselves been seen to be flawed (e.g., Hagen, 1997; Thompson, 1998).”

“…is an extensive literature dealing with conceptual and statistical arguments against overreliance on tests of statistical significance (See, for example, Cohen, 1994; Cortina & Dunlap, 1997; Meehl, 1978; Murphy et al., 2014; Schmidt, 1996), and there are many points of controversy in this debate.”

“Defenses have also appeared in the field’s leading peer-reviewed journals, including American Psychologist (Krueger, 2001, with commentaries) and APA’s quantitative psychology journal Psychological Methods (Frick, 1996; Cortina and Dunlap, 1997; Nickerson, 2000).”

“Cortina and Dunlap (1997), for example, argued that SST can help speak to three interrelated requirements for empirical corroboration, namely, objectivity, exclusion of alternative hypotheses, and exclusion of alternative explanations.”

“There have always been discussions regarding how statistical significance should be interpreted in meta-analyses or more generally for studies with large sample sizes (Cortina and Dunlap, 1997; Cortina and Landis, 2011).”

“…practice that is both logically and practically fallacious, and yet persistent and pernicious in its commonality (for discussions, see Cohen, 1994; Cortina & Dunlap, 1997; Cumming, 2014; Kline, 2004; Meehl, 1990b); p values are the conditional probability of observing a particular test statistic…”

“The technical definition that we will use differs in important ways informal definitions more familiar to physicians, and the difference turns out to be consequential, as witnessed by the existence of a large critical literature dealing with practical and philosophical problems arising from definitions in common use […”

“The contention that the null hypothesis is always false is a common argument against significance testing (Nickerson, 2000; see also Cortina & Dunlap, 1997).”

“Consequently, it does not warrant a unionintersection test of the joint null hypothesis of ‘no positive or negative correlation’ (Cortina and Dunlap [1997], p.”

“…can be agreed upon for making such a decision, and that the conservative stance of an unknown relationship being nil accurately represents resistance to the implementation of a new program or treatment (Abelson, 1997; Chow, 1988; Cortina & Dunlap, 1997; Frick, 1999; Harris, 1997; Wainer, 1999).”

“Finally, although it is not advisable to over interpret the meaningfulness of significance tests for effects that are both small and based on very large samples ( Cortina and Dunlap 1997 ; E den2002), a few comments regarding the similarity of statistical inferences across meta-analyses are warranted.”

“On the other hand, supporters of statistical significance testing (i.e., Cortina and Dunlap, 1997; Robinson and Wainer, 2002) have praised it for what it can do, and have argued that if null hypothesis significance testing (NHST) did not exist, it would have to be invented (Abelson, 1997).”

“…large-sample null results using terminology that is closely associated with the familiar framework of null hypothesis significance testing (NHST; for critical discussion see, e.g., Abelson, 1997; Balluerka, Gómez, & Hidalgo, 2005; Cohen, 1969; Cortina & Dunlap, 1997; Miller, 2017; Nickerson, 2000).”

“Proponents argue that NHSST is useful for certain purposes and is needed for theory corroboration (Chow, 1998; Cortina & Dunlap, 1997; Frick, 1996; Mulaik, Raju, & Harshman, 1997).”

“The use of statistical inference is a subject of current debate (Frick, 1996; Cortina and Dunlap, 1997; Harlow et al., 1997; Chow, 1998; Levin, 1998; Thompson, 1998; Nickerson, 1999, 2000; Wainer, 1999; Howard et al., 2000; Markus, 2001).”

“On this point, Cortina and Dunlap (1997) have commented “we see no point in holding NHST [null hypothesis significance testing] or any other procedure responsible for the shortcomings of the data to which they are applied” (p.”

“When the data are extreme in either tail, the researcher characterizes the evidence as consistent with a population difference in that direction (e.g., μA > μB) because the opposite direction can be rejected along with the null (Cortina & Dunlap, 1997).”

“Meehl (1997, pp. 398–400) shows why this syllogism is inappropriate in appraising SuH, as it is with statistical testing (Cohen, 1994, p. 998; Cortina & Dunlap, 1997, p. 170).”

“Perhaps based on the misguided idea that the 2 approaches lead to similar conclusions (they do have considerable overlap(18)), the blending of these approaches has been stated to adhere to Fisher’s philosophy but NP’s testing procedures.(19) Perezgonzalez(20) provides an excellent tutorial on the uses of both approach…”

“Whereas some (e.g., Cortina & Dunlap, 1997) have argued that replacing NHST by CIs is unlikely to improve scientific practice because both are based on the same information, we suggest that it might.”

“Therefore, it is more interesting to discuss Fisher’s (1954) or Neyman and Pearson’s theories (1933) than NHST. Cumming does so in his book (2012) but not in his article, thus painting an inaccurate picture of the value of data testing in research.”

“…and the Task Force on Statistical Inference (1999), and Krueger (2001)—who are rather Fisherian—and to Kline (2004), Nickerson (2000), Wainer (1999), Cortina and Dunlap (1997), Frick (1996), Cohen (1988, 1994), Schmidt (1992), and Rosnow and Rosenthal (1989)—who are rather Neyman–Pearsonian.”

“…meta-analysis On the other hand, the 10,000 runs of study-replication pairs address replication under a different testing approach, that of Fisher’s, arguably the default one in today’s research (Spielman, 1978; Johnstone, 1986; Cortina and Dunlap, 1997; Hubbard, 2004; Perezgonzalez, 2015b).”

“Cortina and Dunlap (1997) pointed out that such use of the confidence interval to draw a conclusion about the mean is in fact a significance test.”

“The 95% confidence intervals (CIs) [31, 32] of the estimated AIt were calculated by approximating the upper- and lower-half distributions of true population density Xit to respective half-normal distributions, because the distributions of the true population density were skewed and the upper CIs were broader than the…”