Consequences of prejudice against the null hypothesis.

“Greenwald, 1975; Rosenthal, 1984; White, 1982), although Ž ndings are inconsistent. Related to this, the ‘Ž le drawer problem’ (Rosenthal, 1979) refers to the possibility that all published articles are the result of Type I errors, whereas all non-published (i.e. Ž le drawer) studies are the remaining 95%. Reliability…”

“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”

“Meehl (1990a) attributed to the distinguished philosopher Morris Raphael Cohen the saying "All logic texts are divided into two parts.”

“Publication of statistically significant results is more probable (e.g., Greenwald, 1975), increasing the likelihood of Type I errors and an overestimation of the mean effect size in meta-analysis.”

“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47,49,50 .”

“2McGuire (1983a) and Greenwald, Pratkanis, Leippe, and Baumgardner (1986) describe another means to the same end.”

“Note that—as in Experiment 1—this conclusion relies on interpreting a null effect as meaningful, which is possible when samples are adequately powered (Greenwald, 1975).”

“Journal referees may block publication of nonsignificant research findings (Greenwald, 1975).”

“Publication bias has been documented in various disciplines within the biomedical (3, 4, 5, 6, 7, 8, 9) and social sciences (10, 11, 12, 13, 14, 15, 16, 17).”

“In general, one can distinguish between these two alternative accounts of a null finding by considering a different null hypothesis: not that the effect is zero, but that it is very small instead (see e.g., Cohen, 1988, pp. 16-17; Greenwald, 1975; Hodges & Lehmann, 1954; Serlin & Lapsley, 1985).”

“An additional consideration in evaluating meta-analytic methods is the extent to which the magnitude of the physical attractiveness stereotype may have been exaggerated in our review because of a publication bias in favor of significant findings (Greenwald, 1975; Lane & Dunlap, 1978).”

“In summary, the demands for novelty and positive results create incentives for (a) generating new ideas rather than pursuing additional evidence for or against ideas suggested previously; (b) reporting positive results and ignoring negative results (Fanelli, 2012; Greenwald, 1975; Ioannidis & Trikalinos, 2007; Rosenth…”

“…that the predisposition to confirm existing theoretical beliefs is pervasively characteristic of the research behavior of scientists, and I ( Greenwald, 1975a) have reported data showing that psychologists are strongly inclined to disregard research results inconsistent with their…”

“This bias is not trivial: significant findings are estimated to be eight times more likely to be submitted than non-significant ones (Greenwald, 1975), studies with positive findings are around seven times more likely to be published than studies with results supporting the null hypothesis (Coursol…”

“…1993) 54% of the reviewers indicated they would prefer a study with new findings over a replication study.6 When asked what the problems with publishing replications were, comments such as “Not newsworthy” – “Waste of space” “. . . waste of resources” were given (see also Greenwald, 1978).”

“A further difficulty when trying to document the generality of a phenomenon is the likelihood that negative results have more often gone unreported than have rejections of the null hypothesis (Greenwald, 1975).”

“Consider the context, in which positive results are perceived as more publishable than negative results (Greenwald, 1975) and common analytic practices may inadvertently increase the likelihood of obtaining false positives (Simmons et al., 2011).”

“Cook, Gruder, Henningan, and Faly (1979), Greenwald (1975), and Wilson and Shadish (2006) all offer valuable perspectives on broader issues involved in evaluating the veracity of null effects.”

“The nongeneralizability of experimental findings to similar situations and, sometimes, their nonreplicability have been sources of concern (Greenwald, 1975).”

“It has also encouraged researchers to ignore the important cases where there are no differences when conditions are changed (Greenwald, 1975;Rubin, 1989).”

“…than zero, and concluding that it is (basically) zero if we obtain a result that is statistically significantly smaller than that small effect (see, e.g., Cohen, 1988, pp. 16–17; Greenwald, 1975, pp. 16–19; Hodges & Lehmann, 1954; Kraemer, 1983; Rindskopf, 1997; Serlin & Lapsley, 1985, 1993).”

“For instance, Greenwald (1975) found that about half of researchers who produced a rejection of the null hypothesis would submit a report for publication, while only 6 percent who failed to reject the null hypothesis would attempt to publish.”

“Despite their theoretical and empirical value, the existing scientific culture provides few incentives for researchers to conduct replications or report negative results (Greenwald, 1975; Koole & Lakens, 2012).”

“The direction of publication bias is that studies reporting significant results are more often published than studies presenting nonsignificant results (e.g., Greenwald, 1975; Rosnow & Rosenthal, 1989).”

“The reliability of a meta-analysis is sometimes a concern because of the suspicion that some researchers do not report statistically insignificant findings (Greenwald, 1975).”

“The replacement or complementation of NHST with the use of likelihood ratios and the estimation of posterior probabilities by application of Bayes's rule has been proposed by many writers (Edwards, 1965; Edwards et al., 1963; Gelman, Carlin, Stern, & Rubin, 1995; I. J. Good, 1981; Greenwald, 1975; Lindley, 1984; Rinds…”

“Because effect sizes in the published literature are likely to be overestimates of the true effect size (Greenwald, 1975), researchers should err conservatively, toward higher levels of power.”

“Similarly, the significant outcome bias (SOB) has already caused most researchers to no longer even submit non-significant findings (Greenwald, 1975).”

“In addition to publication bias at the journal level, publication bias also likely occurs at the individual scholar level, insofar as scholars decline to submit null results for publication either because such results conflict with beloved theoretical models in which they are invested or simply because they believe nu…”

“A high reproducibility estimate might boost confidence in conventional research and peerreview practices in the face of criticisms about inappropriate flexibility in design, analysis, and reporting that can inflate the rate of false positives (Greenwald, 1975; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simo…”

“This graph of sample size versus effectsize should be symmetrical in the absence of bias, but will tend to be skewed in its presence (Light and Pillemer (1984); see Figs 1 and 2 and further discussion in Section 5.”

“For example, if researchers strongly believe that an effect is real and positive, statistically non-significant or negative estimates of that effect may never be submitted for publication or may be rejected by reviewers and editors (Greenwald, 1975; Sterling, Rosenbaum, & Weinkam, 1995; Rothstein et al., 2006; Ferguso…”

“On a more general level we hope that we have made a case for considering the magnitude of effects in 2D:4D research; merely focusing on the statistical significance of effects may stall progress in 2D:4D research and in other areas [e.g., (Cohen, 1994; Greenwald, 1975; Hunter, 1997; Meehl, 1978)].”

“, nonsignificant results are filed and not even submitted for publication) and would perish (Greenwald, 1975; Rosenthal, 1979).”

“…of research, greater rewards of research support, career advancement, and public recognition tend to fall to the researcher who investigates and confirms a new prediction than to the one who pursues a more familiar idea (see discussions by Armstrong, 1982; Fishman & Neigher, 1982; Greenwald, 1975).”

“This situation, based upon an example first described by Keynes (1921), and again by Ellsberg (1961), demonstrates that two distinct types of uncertainty can be differentiated.”

“…and failure to publish studies with inadequate sample sizes (Newcombe, 1987), development of a more positive attitude toward nonsignificant results (Greenwald, 1975; Rennie & Flanagin, 1992), and improvement of the peer-review and publication processes (Higginson, 1987; Iyengar & Greenhouse, 1988;…”

“Computing the proportional agreement scale values as the average of the respective item values is consistent with how the James et al. (1984) r wg(J) indices are computed.”

“It is commonly assumed that studies that contradict an established hypothesis or support it only weakly and without significance are less likely to be published than studies that support the established hypothesis (Begg & Berlin, 1988; Coursol & Wagner, 1986; Greenwald, 1975; Hedges, 1984).”

“Publication bias can occur at any stage of the publication process where a decision is made [18,19]: in the researcher’s decision to write up a manuscript [20]; in the decision to submit the manuscript to a journal [21,22]); in the decision of journal editors to send a paper out for review [23]; in the reviewer’s reco…”

“…are biased in favor of the publication of statistically significant (p < 0.05) results, thus contaminating their respective literatures with Type I errors whose incidence exceeds that prescribed by nominal alpha levels such as 0.05 (Feige, 1975; Greenwald, 1975; Hubbard and Armstrong, 1992).”

“As explained previously, published samples tend to report more significant and positive effect sizes than the literature as a whole (e.g., Greenwald, 1975; Hopewell, McDonald, Clarke, & Egger, 2007; Song et al., 2010).”

“…such as selective reporting, presenting exploratory findings as confirmatory tests, and failures of documentation, transparency, and sharing (Casadevall and Fang, 2012; Gelman and Loken, 2013; Greenwald, 1975; Kimmelman et al., 2014; Makel et al., 2012; Nosek et al., 2012; Rosenthal, 1979).”

“Although it can be suggested that journal publication is biased towards studies that show effects, Greenwald (1975) argues that this does not invalidate the conclusions drawn.”

“In any case, because of the focus on rejecting nulls with frequentist methods, theorizing null effects is almost never done, even though hypothesizing nulls can be useful (Greenwald, 1975; Kruschke, 2011; Schwab et al., 2011).”

“These include editors and reviewers selectively rejecting manuscripts with negative results (reviewer bias; Greenwald, 1975; Mahoney, 1977) and researchers deciding not to submit studies with negative results for publication (file-drawering; Rosenthal, 1979).”

“Greenwald (1975) recommends testing a null range—rather than a null point to demonstrate why a .05 significance test warrants acceptance of the null hypothesis.”

“, nonsignificant) results are much more likely to end up in the file drawer (Greenwald, 1975).”

“…results of Study 2 therefore shift the burden of proof to those who characterize ideologically patterned attributions as rooted in underlying stable dispositional differences between liberals and conservatives (see Greenwald, 1975, on the power of null hypothesis results to shift burdens of proof).”

“Note that the logic of NHST demands that the equal signs in Equation 3 mean equal to an infinite number of decimal places . If one weakens this requirement such that "=" means "pretty much equal," then one must add additional assumptions specifying what is meant by "pretty much." Although the mathematical machinery fo…”

“…more likely to follow-up on initial studies that yielded higher e ff ect size estimates than initial studies that yielded lower e ff ect size estimates (cf. Greenwald, 1975), simply because these studies require less resources to observe a statistically signi fi cant result in the expected direction.”

“Due to concerns regarding the publication bias against null findings (e.g., Greenwald, 1975; Hubbard & Armstrong, 1997), we also compared effect sizes across published and unpublished studies, using journal impact factors to compare high-and low-impact publications.”

“, 2004; Coursol and Wagner, 1986) and selective submission of papers (Franco et al., 2014; Greenwald, 1975), as well as editors’ and reviewers’ biases (although some empirical work suggests that the latter may be a weaker influence than author decision making; see Lee et al.”

“Most notably, results of published data are generally more likely to reject the null hypothesis than unpublished data testing the same notion (Greenwald, 1975) and are more likely to exhibit larger effect sizes (Lipsey & Wilson, 1993).”