Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract
There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST).
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Sterling, 1959
The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them.
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
Positive results are more likely to be published than negative results.
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
These causes of overestimation can dramatically increase Type I error rates in the meta-analysis.
“These causes of overestimation can dramatically increase Type I error rates in the meta-analysis (Sterling, 1959).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Sterling, 1959
Cited by
Sorted by citations
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”
A Note on the Theme of Too Many InstrumentsPreprint
Roodman, G. D. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1101731 · Cited by 5,829 · see citing works
Systematic Reviews in the Social Sciences: A Practical Guide
Beelmann, A., Petticrew, M., & Roberts, H. (2005) · European Psychologist · doi:10.1027/1016-9040.11.3.244 · Cited by 5,231 · see citing works
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47,49,50 .”
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
“Publication bias has been documented in various disciplines within the biomedical (3, 4, 5, 6, 7, 8, 9) and social sciences (10, 11, 12, 13, 14, 15, 16, 17).”
BackgroundP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
“Scientists have been grappling with the problem of publication bias for at least 50 years (Sterling, 1959).”
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
“, “positive”) results, and there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [4–10].”
Background<i>meta-analysis</i> bias in location and selection of studies
Egger, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7124.61 · Cited by 1,374 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
“This appears to have been true for more than 50 years (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · see citing works
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · see citing works
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Field, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
“…than non-significant ones (Greenwald, 1975), studies with positive findings are around seven times more likely to be published than studies with results supporting the null hypothesis (Coursol & Wagner, 1986), and 97% of articles in psychology journals report significant results (Sterling, 1959).”
ResultWriting meta-analytic reviews.
Rosenthal, R. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.118.2.183 · Cited by 1,108 · see citing works
“file drawers (Sterling, 1959).”
BackgroundNull hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · see citing works
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
“For example, one indication that bias influences the research process is that researchers seek to confirm, not falsify, their main hypothesis (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
A meta-analytic review of prolonged exposure for posttraumatic stress disorder
Powers, M. B., Halpern, J. M., Ferenschak, M. P., Gillihan, S. J., & Foa, E. B. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2010.04.007 · Cited by 1,005 · see citing works
“Publication bias: “the file drawer problem” Several authors suggest there may be a potential discrepancy between the number of published trials and the total number that are completed (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959).”
BackgroundStatistical Procedures and the Justification of Knowledge in Psychological Science
Rosnow, R., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 951 · see citing works
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
“…from the preferential reporting of statistically significant or ‘positive’ scientific results has long been a focus and concern of statisticians (Sterling 1959; Rosenthal 1979; Hedges and Olkin 1985; Begg and Berlin 1988; Sterling, Rosenbaum, and Weinkam 1995; Copas 1999; Senn 2008; Mandel…”
BackgroundVirtual reality exposure therapy for anxiety disorders: A meta-analysis
Powers, M. B., & Emmelkamp, P. M. G. (2008) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2007.04.006 · Cited by 904 · see citing works
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · see citing works
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
“Psychology has had a replicability problem for decades (Sterling, 1959), but it seems as if the time has finally come to improve scientific practices in psychology (Schimmack, 2012).”
BackgroundThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
“It has long been argued that the combined outcomes of NHST in the scientific literature are too good to be true (Fanelli, 2010; Fiedler, 2011; Ioannidis, 2008a; Sterling, 1959; Vul, Harris, Winkielman, & Pashler, 2009).”
BackgroundToward an Integrated Theory of Schizotaxia, Schizotypy, and Schizophrenia
Meehl, P. (1990) · Journal of Personality Disorders · doi:10.1521/pedi.1990.4.1.1 · Cited by 792 · see citing works
Cheung, A. C. K., & Slavin, R. E. (2013) · Educational Research Review · doi:10.1016/j.edurev.2013.01.001 · Cited by 792 · see citing works
“Sixth, in contrast to some earlier reviews (Niemiec et al., 1987; Smith, 1980; Sterling, 1959), we found no statistically significant difference between published articles and unpublished reports.”
Result<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · see citing works
“Scientific journals usually do not publish results unless they are statistically significant (henceforth assumed to correspond to p ≤ .05), a fact we will refer to as publication bias (see e.g., Fanelli, 2012; Rosenthal, 1979; Sterling, 1959).”
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
“As a consequence, replications (Makel, Plucker, & Hegarty, 2012) and negative results (Fanelli, 2010; Sterling, 1959) are rare in the published literature.”
BackgroundPublication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Peters, G.-J. Y., Ruiter, R. A. C., & Kok, G. (2013) · Health Psychology Review · doi:10.1080/17437199.2012.703527 · Cited by 679 · see citing works
“In 1959, Sterling (1959) found that 97% of the hypothesistesting studies reported significant findings, and 30 years later, this percentage had only decreased slightly to 95% (Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundExercise Interventions for Mental Health: A Quantitative and Qualitative Review.
Stathopoulou, G., Powers, M. B., Berry, A. C., Smits, J. A. J., & Otto, M. W. (2006) · Clinical Psychology: Science and Practice · doi:10.1111/j.1468-2850.2006.00021.x · Cited by 663 · see citing works
Olatunji, B. O., Davis, M. L., Powers, M. B., & Smits, J. A. J. (2013) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2012.08.020 · Cited by 659 · see citing works
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634 · see citing works
“…Reporting Bias For at least a half century, ‘publication bias’ has been widely recognised as a potential threat to the validity of empirical science (Sterling, 1959; Tullock, 1959; Rosenthal, 1979; Glass et al., 1981; Lovell, 1983; Hedges and Olkin, 1985; Begg and Berlin, 1988; De Long and Lang,…”
BackgroundAcceptance and commitment therapy: A meta-analytic review
Powers, M. B., Zum Vörde Sive Vörding, M. B., & Emmelkamp, P. M. G. (2009) · Psychotherapy and Psychosomatics · doi:10.1159/000190790 · Cited by 619 · see citing works
From Tools to Theories: A Heuristic of Discovery in Cognitive Psychology.
Gigerenzer, G. (1991) · Psychological Review · doi:10.1037/0033-295x.98.2.254 · Cited by 619 · see citing works
“By 1955, more than 80% of the experimental articles in leading journals used inferential statistics to justify conclusions from the data (Sterling, 1959).”
BackgroundA meta-analytic review of prospective memory and aging.
Henry, J., MacLeod, M., Phillips, L., & Crawford, J. (2004) · Psychology and Aging · doi:10.1037/0882-7974.19.1.27 · Cited by 616 · see citing works
“Particularly problematic is “the file drawer problem” (Rosenthal, 1979), which refers to the fact that significant results are more likely to be published than nonsignificant results (Easterbrook, Berlin, Gopalan, & Mathews, 1991; Sterling, 1959).”
ResultFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
“…(e.g., Bakker, van Dijk, & Wicherts, 2012; Bozarth & Roberts, 1972; Fanelli, 2010, 2012; Pigott, Valentine, Polanin, Williams, & Canada, 2013; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), low reliability among peer reviewers (e.g., Cicchetti, 1991; Cole, 1992; D. Peters & Ceci,…”
BackgroundInfluentialStanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605 · see citing works
“…empirical effect, the probability that a majority vote-count comes to the wrong conclusion increases with added research.3 Other areas of social science widely acknowledge the importance of publication selection and its associated bias (Sterling, 1959; Rosenthal, 1979; Begg and Berlin, 1988).”
BackgroundReplicability, Robustness, and Reproducibility in Psychological Science.Preprint
Nosek, B., Hardwicke, T., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Struhl, M. K., Nuijten, M., et al. (2020) · Annual Review of Psychology · doi:10.31234/osf.io/ksfvq · Cited by 592 · see citing works
Meta‐analysis of transmitter effects on avian behaviour and ecology
Barron, D. G., Brawn, J., & Weatherhead, P. J. (2010) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2010.00013.x · Cited by 575 · see citing works
“If studies with non-significant results tend to go unpublished, a reporting bias known as the ‘file-drawer effect’ would result (Sterling 1959).”
ResultUnder what conditions does theory obstruct research progress?
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.216 · Cited by 574 · see citing works
“…customary .05, to combat the effective "alpha inflation" that is incurred by repeated serial testing of the same prediction (Selvin & Stuart, 1966; Sterling, 1959 ); (b) standardly reporting measures of magnitude of effect, because null-hypothesis rejection can be obtained with trivially small…”
BackgroundWhy Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
“However, it is striking that the proportion of significant results in psychological science is approaching 100%, and this has been noticed even half a century ago (Sterling, 1959) with no change in the interim (Bakker et al., 2012; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundA brief history of research synthesis
Chalmers, I., Hedges, L. V., & Cooper, H. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001003 · Cited by 552 · see citing works
“A study of reports published in a sample of psychology journals published in the late 1950s revealed that a very high proportion reported statistically significant results (Sterling, 1959).”
BackgroundNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Dickersin, K. (2003) · JAMA · doi:10.1001/jama.290.4.516 · Cited by 512 · see citing works
“Fordecades, scientists have complained about the overreportingof“positive”results(results that favor thenewtherapy).(27) Publicationbias is common and is almost always because of failure of the investigators to complete, write up, and submit to journals findings from trials with negative results.”
BackgroundVote-counting methods in research synthesis.
Hedges, L., & Olkin, I. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.359 · Cited by 511 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · see citing works
Stanley, T. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00250.x · Cited by 489 · see citing works
Reducing implicit racial preferences: I. A comparative investigation of 17 interventions.
Lai, C., Marini, M., Lehr, S. A., Cerruti, C., Shin, J., Joy-Gaba, J. A., Ho, A. K., Teachman, B., Wojcik, S., Koleva, S. P., et al. (2014) · Journal of experimental psychology. General · doi:10.1037/a0036260 · Cited by 482 · see citing works
Sex differences in the ability to recognise non-verbal displays of emotion: A meta-analysis
Thompson, A. E., & Voyer, D. (2014) · Cognition and Emotion · doi:10.1080/02699931.2013.875889 · Cited by 476 · see citing works
“This problem is commonly known as the file-drawer problem (Sterling, 1959).”
BackgroundTheory of mind and prosocial behavior in childhood: A meta-analytic review.
Imuta, K., Henry, J. D., Slaughter, V., Selcuk, B., & Ruffman, T. (2016) · Developmental Psychology · doi:10.1037/dev0000140 · Cited by 474 · see citing works
“Particularly problematic is “the file drawer problem” (Rosenthal, 1979), which refers to the fact that significant results are more likely to be published than nonsignificant results (Easterbrook, Gopalan, Berlin, & Matthews, 1991; Ferguson & Heene, 2012; Sterling, 1959).”
ResultPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · see citing works
“Empirical research using various methodologies, and conducted in diverse substantive areas, suggests that statistically significant results are more likely than nonsignificant results to be published and presented (Bozarth & Roberts, 1972; Greenwald, 1975; Smart, 1964; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1…”
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
“…number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959).”
BackgroundResultInfluentialSterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
“The suggestion that published results of scientific investigations are not a representative sample of results from all scientific studies was first made in 1959 (Sterling 1959).”
BackgroundMethodSelf-citationReporting bias in medical research - a narrative review
McGauran, N., Wieseler, B., Kreis, J., Schüler, Y.-B., Kölsch, H., & Kaiser, T. (2010) · Trials · doi:10.1186/1745-6215-11-37 · Cited by 452 · see citing works
Replications and Extensions in Marketing – Rarely Published But Quite Contrary
Hubbard, R., & Armstrong, J. (1994) · International Journal of Research in Marketing · doi:10.1016/0167-8116(94)90003-5 · Cited by 444 · see citing works
“Sterling (1959) did not discover a single replication in his inspection of 362 published psychology articles.”
BackgroundMeta-Analyses of the Effects of Media Images on Men's Body-image Concerns
Barlett, C. P., Vowels, C. L., & Saucier, D. (2008) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2008.27.3.279 · Cited by 435 · see citing works
“The fail–safe N estimates the number of studies with null results that need to exist in order to make the overall effect size nonsignificant (Sterling, 1959).”
BackgroundPublication Bias: The Problem That Won't Go Away
DICKERSIN, K., & MIN, Y. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26343.x · Cited by 427 · see citing works
Tryon, W. (2001) · Psychological methods · doi:10.1037/1082-989x.6.4.371 · Cited by 423 · see citing works
“(cf. Bakan, 1966; Berkson, 1942; Carver, 1978, 1993; Chow, 1998; Cohen, 1994; Dar, Serlin, & Omer, 1994; Hagen, 1997; Harlow, 1997; Hodges & Lehmann, 1954; Hogben, 1957; Hunter & Schmidt, 1990; Lykken, 1968; Meehl, 1967, 1978, 1990a, 1990b; Morrison & Henkel, 1970; Rozeboom, 1960; Sterling, 1959) and culminated in a s…”
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410 · see citing works
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 406 · see citing works
Sex differences in visual-spatial working memory: A meta-analysis
Voyer, D., Voyer, S. D., & Saint-Aubin, J. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1085-7 · Cited by 400 · see citing works
“This problem, called the Bfile-drawer problem^ (Sterling, 1959) has potential to bias meta-analytic results.”
BackgroundMcShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
“These concerns date back centuries (Boyle, 1661/1965; Dickersin, 2005; Editors, 1909; Ferriar & Simmons, 1792; Hall, 1959; Lane & Dunlap, 1978; Sterling, 1959) and relate not only to issues surrounding the size, direction, and statistical significance of study results but also to issues surrounding the availability an…”
BackgroundResultInfluentialMisinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391 · see citing works
“The result is publication bias, whereby the published literature provides a biased sample of the research actually conducted (54).”
BackgroundRosenthal, R. (1991) · Psychosomatic Medicine · doi:10.1097/00006842-199105000-00001 · Cited by 380 · see citing works
“Both behavioral researchers and statisticians have long suspected that the studies published in the behavioral and social sciences are a biased sample of the studies that are actually carried out (69-72).”
BackgroundA general linear model for estimating effect size in the presence of publication bias
Vevea, J. L., & Hedges, L. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 367 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 359 · see citing works
“…using different selection criteria for articles and hypotheses have found very similar rates of supported hypotheses in the literature: 97.28% in Sterling (1959), 95.56% in Sterling et al. (1995), and 97% in the original studies included in the Reproducibility Project: Psychology (Open Science…”
BackgroundResultInfluentialEyding, D., Lelgemann, M., Grouven, U., Härter, M. C., Kromp, M., Kaiser, T., Kerekeş, M., Gerken, M., & Wieseler, B. (2010) · BMJ · doi:10.1136/bmj.c4737 · Cited by 591 · see citing works
Questionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
“The bias exists in many sciences [22], has been documented for decades in some disciplines (e.g., in psychology, see Sterling, 1959 [23]) and may be getting stronger across science, with a detectable increase in the proportion statistically significant results over the last 25 years [24].”
BackgroundMeta-analysis of genetic association studies.
Munafo, M., & Flint, J. (2004) · Trends in Genetics · doi:10.1016/j.tig.2004.06.014 · Cited by 346 · see citing works
A reevaluation of the absenteeism-job satisfaction relationship
Hackett, R. D., & Guion, R. M. (1985) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(85)90028-7 · Cited by 315 · see citing works
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K., Witteloostuijn, A., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 337 · see citing works
Diabetes mellitus and risk of prostate cancer: a meta-analysis
Bonovas, S., Filioussi, K., & Tsantes, A. (2004) · Diabetologia · doi:10.1007/s00125-004-1415-6 · Cited by 332 · see citing works
“Existence of a bias in favour of publication of statistically significant results is well documented in the literature [41, 42, 43].”
ResultDoes psi exist? Replicable evidence for an anomalous process of information transfer.
Bem, D. J., & Honorton, C. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.4 · Cited by 330 · see citing works
Factors associated with failure to publish large randomized trials presented at an oncology meeting.
Krzyzanowska, M., Pintilie, M., & Tannock, I. (2003) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.290.4.495 · Cited by 325 · see citing works
Meta-Analysis of Theory of Mind and Peer Popularity in the Preschool and Early School Years
Slaughter, V., Imuta, K., Peterson, C. C., & Henry, J. D. (2015) · Child Development · doi:10.1111/cdev.12372 · Cited by 324 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · see citing works
A systematic review of educational digital storytelling
Wu, J. J., & Chen, D.-T. V. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103786 · Cited by 415 · see citing works
Examining the Effectiveness of Solution-Focused Brief Therapy: A Meta-Analysis
Kim, J. S. (2008) · Research on Social Work Practice · doi:10.1177/1049731507307807 · Cited by 317 · see citing works
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312 · see citing works
“Across disciplines, most published studies demonstrate positive results—results that indicate an expected association between variables or a difference between experimental conditions ( Sterling, 1959 ; Fanelli, 2010 , 2012 ).”
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307 · see citing works
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
“, significant) results a near necessity for publication (Fanelli, 2010, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundZwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, M. B. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 474 · see citing works
Behavioral couples therapy (BCT) for alcohol and drug use disorders: A meta-analysis
Powers, M. B., Vedel, E., & Emmelkamp, P. M. G. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.02.002 · Cited by 300 · see citing works
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
“However, a is always fixed independently of what p value we find in an experiment whereas p values can be considered random variables, varying widely from experiment to experiment (Murdoch et al. 2008; Hung et al. 1997; Simonsohn et al. 2014a,b; Sterling 1959).”
BackgroundA Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · see citing works
“Nearly six decades ago, Sterling [8] noted this 48 anomaly in the public record: In four prominent psychology journals, 95 to 99% of 49 studies that performed a significance test rejected the null hypothesis (i.”
The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 293 · see citing works
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 289 · see citing works
Rutledge, T., & Hogan, B. E. (2002) · Psychosomatic Medicine · doi:10.1097/01.psy.0000031578.42041.1c · Cited by 286 · see citing works
“Due to the well-known underreporting of null effects (34, 35), we computed fail-safe n values for the overall and categorized psychological effects to estimate the statistical tolerance for existing “file drawer” or future negative findings (33).”
Method
- Authors
- Sterling, T. D.
- Affiliations
- University of Cincinnati (US)
- Corresponding author
- Sterling, Theodore D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1959-03
- Volume
- 54
- Issue
- 285
- Pages
- 30-34
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2062908954
- Cited by
- 964
- References
- 0
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
