The DOI matches its Crossref record.
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract
There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Sterling, 1959
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“These causes of overestimation can dramatically increase Type I error rates in the meta-analysis (Sterling, 1959).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Sterling, 1959
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- 1958, Cancer Research 18
- 1957, Publication Manual,
- Edwards A. L., 1950, Experimental Design in Psychological Research
- Lewis D., 1949, Psychological Bulletin, 46, 433, 10.1037/h0059088 · DOI
- Lubin A., 1957, American Psychologist, 8, 519, 10.1037/h0039746 · DOI
- McNemar, 1955, Psychological Statistics,
- Savage L. J., 1954, The Foundations of Statistics
- Walker H. M., 1947, Elementary Statistical Methods
Cited by
Sorted by citations
Items 301–400 of 888
Zhang, L. (2007) · doi:10.1353/rhe.2007.0057 · Cited by 51
“results (Light & Pillemer, 1984; Rosenthal, 1979; Sterling, 1959), and I suspect that studies with theory-consistent results may also receive more favorable consideration.”
ResultDo we know the effectiveness of behavioural interventions?
Aral, S., & Peterman, T. (1998) · The Lancet · doi:10.1016/s0140-6736(98)90010-1 · Cited by 51
A power‐analytic examination of contemporary communication research
Chase, L. J., & Tucker, R. K. (1975) · Speech Monographs · doi:10.1080/03637757509375874 · Cited by 49
An Analysis of the Use of Statistical Testing in Communication Research
Katzer, J., & Sodt, J. (1973) · Journal of Communication · doi:10.1111/j.1460-2466.1973.tb00947.x · Cited by 49
The benefits of preregistration and Registered Reports
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2376046 · Cited by 93
The Logic of Tests of Significance
Spielman, S. (1974) · Philosophia Scientiæ · doi:10.1086/288590 · Cited by 50
“Sterling, [8], where Sterling shows that 81% of the papers published in 1955 in four leading journals of psychology reported the outcomes of tests of significance.”
BackgroundEvaluating deception in psychological research.
Stricker, L. J., Messick, S., & Jackson, D. N. (1969) · Psychological Bulletin · doi:10.1037/h0027232 · Cited by 48
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F. F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231187988 · Cited by 45
“The social sciences do not have an extensive registration infrastructure, so selective reporting has mainly been studied by comparing publications with dissertations (Cairo et al., 2020; Mazzola & Deuling, 2013; O’Boyle et al., 2017) and archived research proposals (Franco et al., 2016).”
Social incentive factors in interventions promoting sustainable behaviors: A meta-analysis
Phu Nguyen-van, B., Stenger, A., & Tiet, T. T. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260932 · Cited by 55
Potential publication bias in the stereotype threat literature: Comment on Nguyen and Ryan (2008).
Zigerell, L. J. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000188 · Cited by 49
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48
Clayson, P., Carbine, K. A., & Larson, M. (2019) · International Journal of Psychophysiology · doi:10.31234/osf.io/hj6zc · Cited by 48
Publication bias in “Red, rank, and romance in women viewing men,” by Elliot et al. (2010).
Francis, G. (2013) · Journal of Experimental Psychology General · doi:10.1037/a0027923 · Cited by 41
“Although many of the ideas about too many rejections of the null hypothesis have been discussed previously (Sterling, 1959; Sterling, Rosenbaum & Weinkam, 1995), it seems that experimental psychologists generally misunderstand how replication should function in a”
BackgroundBones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 51
“Sterling (1959) first documented psychologists’ remarkable precognitive capacities.”
BackgroundScientific Publishing in Biomedicine: A Brief History of Scientific Journals
Ghasemi, A., Mirmiran, P., Kashfi, K., & Bahadoran, Z. (2022) · International Journal of Endocrinology and Metabolism · doi:10.5812/ijem-131812 · Cited by 47
A meta-analysis of contingent-capture effects
Büsel, C., Voracek, M., & Ansorge, U. (2018) · Psychological Research · doi:10.1007/s00426-018-1087-3 · Cited by 47
“Publication bias is a long-known problem in psychology in general (Rosenthal, 1979; Sterling, 1959), and research Contingent−capture effects trim and fill”
BackgroundWandall, B., Hansson, S. O., & Rudén, C. (2007) · Archives of Toxicology · doi:10.1007/s00204-007-0194-5 · Cited by 44
“…developed for similar studies in other areas, namely (a) for published studies, comparing the number of hypothesis-testing studies reporting positive results, i.e., a rejection of the null-hypothesis, to the number of studies without positive results (see Sterling 1959 for such an investigation).”
ResultThe p-value Function and Statistical Inference
Fraser, D. (2019) · American Statistician · doi:10.1080/00031305.2018.1556735 · Cited by 46
“Quite early, Sterling (1959) discussed the use of this modification for journal article acceptance, and documented serious risks.”
BackgroundToo Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra Psychology · doi:10.1525/collabra.71 · Cited by 45
Discovery in Cognitive Psychology: New Tools Inspire New Theories
Gigerenzer, G. (1992) · Science in Context · doi:10.1017/s0269889700001216 · Cited by 46
Publication selection in health policy research: The winner's curse hypothesis
Costa‐Font, J., McGuire, A. J., & Stanley, T. D. (2012) · Health Policy · doi:10.1016/j.healthpol.2012.10.015 · Cited by 41
“…For a half-century, medical researchers, as well as some economists have expressed concerns about the detrimental effects of publication selection (Sterling, 1959; Tullock, 1959; Feige, 1975; Rosenthal, 1979; Lovell, 1983; Begg and Berlin, 1988; DeLong and Lang, 1992; Card and Krueger, 1995;…”
BackgroundResult-Blind Peer Reviews and Editorial Decisions
Greve, W., Bröder, A., & Erdfelder, E. (2013) · European Psychologist · doi:10.1027/1016-9040/a000144 · Cited by 46
“Publication bias in particular was already bemoaned and documented in the 1950s (Sterling, 1959), but there was little change in editorial publication policy in the decades that followed (see, e.”
BackgroundAn Investigation of Positive-Findings Bias in Evaluation of Organization Development Interventions
Woodman, R., & Wayne, S. (1985) · Academy of Management Journal · doi:10.2307/256243 · Cited by 45
“Terpstra (1981) and others have suggested the need to discover the causes of evaluation bias.”
BackgroundMethodInfluentialLogical fallacies in animal model research
Sjoberg, E. A. (2017) · Behavioral and Brain Functions · doi:10.1186/s12993-017-0121-8 · Cited by 54
Classes and Castles: Impact of Social Stratification on Housing Inequality in Post-Socialist States
Lux, M., Sunega, P., & Katrňák, T. (2013) · European Sociological Review · doi:10.1093/esr/jcr060 · Cited by 44
An Investigation of the Prevalence of Replication Research in Human Factors
Jones, K. S., Derby, P. L., & Schmidlin, E. A. (2010) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810384394 · Cited by 41
“, psychology) suggest that replications are also uncommon in that domain (Bozarth & Roberts, 1972; Mahoney, 1985; Schmidt, 2009; Sterling, 1959; cf. Neuliep & Crandall, 1993a).”
BackgroundInfluentialBonovas, S., Filioussi, K., & Sitaras, N. (2005) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2005.02386.x · Cited by 44
What exercise and sport scientists don't understand.
Speed, H., & Andersen, M. B. (2000) · Journal of Science and Medicine in Sport · doi:10.1016/s1440-2440(00)80051-1 · Cited by 44
Emotional intelligence and romantic relationship satisfaction: A systematic review and meta-analysis
Jardine, B. B., Vannier, S. A., & Voyer, D. (2022) · Personality and Individual Differences · doi:10.1016/j.paid.2022.111713 · Cited by 62
Data management and sharing: Practices and perceptions of psychology researchers
Borghi, J., & Gulick, A. E. V. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252047 · Cited by 46
“Interrelated issues including an overrepresentation of positive results [1, 2] unreported flexibility in analytical methods [3–5], and low levels of statistical power [6, 7] have resulted in uncertainty about the robustness of results described in the psychology literature.”
BackgroundComparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov
Gandhi, R. A., Jan, M., Smith, H. N., Mahomed, N. N., & Bhandari, M. (2011) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-12-278 · Cited by 46
“Improving instructions for authors submitting manuscripts to journals, and making trial registration a necessity is an essential step to move towards reliable reporting of clinical trial results [33,34].”
BackgroundPositive-outcome bias: comparison of emergency medicine and general medicine literatures.
Moscati, R., Jehle, D., Ellis, D. G., Fiorello, A., & Landi, M. K. (2008) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.1994.tb02443.x · Cited by 43
A Parsimonious Weight Function for Modeling Publication Bias
Citkowicz, M., & Vevea, J. L. (2017) · Psychological methods · doi:10.1037/met0000119 · Cited by 42
The impact of bilingualism on cognitive ageing and dementia
Bak, T. H. (2016) · Linguistic Approaches to Bilingualism · doi:10.1075/lab.15002.bak · Cited by 105
Perneger, T., & Combescure, C. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.003 · Cited by 51
Halligan, S., & Altman, D. (2007) · Radiology · doi:10.1148/radiol.2431051823 · Cited by 41
Büttner, F., Toomey, E. C., McClean, S., Roe, M., & Delahunt, E. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101863 · Cited by 56
Giner‐Sorolla, R. (2018) · European Review of Social Psychology · doi:10.1080/10463283.2018.1542902 · Cited by 52
The cult of statistical significance
Krämer, W. (2010) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-8037 · Cited by 1
Adjusting for publication bias: modelling the selection process.
Preston, C., Ashby, D., & Smyth, R. (2004) · Journal of Evaluation In Clinical Practice · doi:10.1111/j.1365-2753.2003.00457.x · Cited by 40
Statistical Power: Derivation, Development, and Data-Analytic Implications
Chase, L. J., & Tucker, R. K. (1976) · The Psychological Record · doi:10.1007/bf03394413 · Cited by 40
HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine LearningPreprint
Gencoglu, O., van Gils, M., Guldogan, E., Morikawa, C., Süzen, M., Gruber, M., Leinonen, J., & Huttunen, H. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07633 · Cited by 40
“Publication bias , the phenomenon occurring when the probability of a scientific study being published is not independent of its results [18], leads to systematic difference in the findings of published tests of a claim from the findings of all tests of the same claim [19].”
BackgroundUsing meta-analysis for evidence synthesis: The case of incomplete neutralization in German
Nicenboim, B., Roettger, T. B., & Vasishth, S. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.06.001 · Cited by 45
Rosenberger, R., & Stanley, T. (2006) · Ecological Economics · doi:10.1016/j.ecolecon.2006.03.018 · Cited by 39
“Medical researchers and many areas of social science have long recognized the seriousness of publication selection (Sterling, 1959; Rosenthal, 1979; Begg and Berlin, 1988), and more recently, economists have uncovered publication bias in many areas of economic research with the help of meta- Table…”
Negative results in computer vision: A perspective
Borji, A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.10.001 · Cited by 37
“Publication bias: coined by Theodore Sterling in 1959 [41], points to the situation where ”publication of research results depends not just on the quality of the research”
On the evidence for narrow, relativistically shifted X-ray lines
Vaughan, S., & Uttley, P. (2008) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2008.13772.x · Cited by 38
“See Sterling (1959), Rosenthal (1979) and Begg & Berlin (1988) for general discussion of publication bias, and also Stern & Simes (1997) and Naylor (1997) for a more recent discussion of the importance of publication bias in the context of medical trials.”
BackgroundSchwab, S., Giuachin, K., & Held, L. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045942 · Cited by 44
Replication crisis or an opportunity to improve scientific production?
Navarro, M. D. F., Pascual‐Llobell, J., Pascual‐Soler, M., Perezgonzalez, J. D., & Berríos-Riquelme, J. (2020) · European Journal of Education · doi:10.1111/ejed.12417 · Cited by 55
Retirement and Health Outcomes in a Meta-Analytical FrameworkPreprint
Filomena, M., & Picchio, M. (2022) · Social Science Research Network · doi:10.2139/ssrn.3900864 · Cited by 36
“For example, psychological and medical research has long acknowledged it since the end of the 1950s (Sterling, 1959; Rosenthal, 1979; Begg and Berlin, 1988).”
BackgroundSwedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 471
Kosara, R., & Haroz, S. (2018) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv.2018.8634392 · Cited by 27
“The results of these studies need to be published whether they agree with the initial one or not, otherwise they lead to the file-drawer problem: studies whose results do not reach the significance threshold are not published [25].”
BackgroundTest–Retest Reliability in Infant Speech Perception Tasks
Cristia, A., Seidl, A., Singh, L., & Houston, D. (2016) · Infancy · doi:10.1111/infa.12127 · Cited by 36
“…the resulting literature is intrinsically biased, and it will contain many more false positives than if we valued experiments based on their methods (Sterling, 1959) — and, some years ago, this bias was found to be even greater in psychological than medical journals (Sterling, Rosenbaum & Weinkam,…”
Publication Bias in Methodological Computational Research
Boulesteix, A., Stierle, V., & Hapfelmeier, A. (2015) · Cancer Informatics · doi:10.4137/cin.s30747 · Cited by 35
“Moreover, we see at least two additional issues: (1) researchers might declare their research outcome a “success” by setting a low threshold on this ordinal or continuous scale and (2) the success may have been generated artificially, either wittingly or unwittingly, for example, by selecting the settings, the dataset…”
BackgroundYu, D., Hu, J. S., Yan, H., Yang, X., Yang, J.-Y., & Shen, H. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-297 · Cited by 39
The Replication Problem in Sociology: A Report and a Suggestion*
Wilson, F. D., Smoke, G. L., & Martin, J. (1973) · doi:10.1111/j.1475-682x.1973.tb00711.x · Cited by 36
Linguistic prosody in autism spectrum disorder—An overview
Grice, M., Wehrle, S., Krüger, M., Spaniol, M., Cangemi, F., & Vogeley, K. T. (2023) · Language and Linguistics Compass · doi:10.1111/lnc3.12498 · Cited by 44
When Does Partitioned Pricing Lead to More Favorable Consumer Preferences?: Meta-Analytic Evidence
Abraham, A. T., & Hamilton, R. W. (2018) · Journal of Marketing Research · doi:10.1177/0022243718800724 · Cited by 45
Campbell, L., Loving, T. J., & LeBel, E. P. (2014) · Personal Relationships · doi:10.1111/pere.12053 · Cited by 40
“Sterling, T. D., Rosebaum,W. L., &Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa.”
BackgroundCreativity is more than novelty: Reconsidering replication as a creativity act.
Makel, M. C., & Plucker, J. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035811 · Cited by 35
Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.2307/1169648 · Cited by 35
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 43
“These causes of overestimation can dramatically increase Type I error rates in the metaanalysis (Sterling, 1959).”
BackgroundEdelsbrunner, P. A., & Thurn, C. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100590 · Cited by 53
A 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression
O’Boyle, E. H., Banks, G. C., Carter, K., Walter, S. L., & Yuan, Z. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9539-8 · Cited by 42
“However, statistically significant MMR tests may be more likely to be published than those that fail to reach significance (Greenwald, 1975; Orlitzky, 2012; Porter, 1992; Rosenthal, 1979; Sterling, 1959).”
BackgroundWhich findings should be published?
Kasy, M., & Frankel, A. (2018) · American Economic Journal: Microeconomics · doi:10.31222/osf.io/mbvz3 · Cited by 34
Miller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 41
“In addition, the tendency of journals to publish only statistically significant results is known to increase the frequency of FPs within the published literature (e.g., Bakker, Van Dijk, & Wicherts, 2012; Fanelli, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundThe Effects of Computer Based Instruction in Elementary Schools: A Quantitative Synthesis.
Niemiec, R. (1987) · Journal of Research on Computing in Education · doi:10.1080/08886504.1987.10781826 · Cited by 34
Brand, J. (2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781412952644.n170 · Cited by 34
“That probability was significantly higher when the child placed the dot in area A (25; 95% CI [22, 28]) rather than B (10; 95%CI [8, 12]), t (967) = 9.”
BackgroundPublication bias in meta-analyses of the efficacy of psychotherapeutic interventions for depression.
Niemeyer, H., Musch, J., & Pietrowsky, R. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0031152 · Cited by 35
Chalmers, I., & Dickersin, K. (2013) · F1000Research · doi:10.12688/f1000research.2-1.v1 · Cited by 35
The insignificance of statistical significance
Aarts, S., Winkens, B., & van den Akker, M. (2012) · European Journal of General Practice · doi:10.3109/13814788.2011.618222 · Cited by 33
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 42
“There is ample and long-standing evidence across disciplines that “statistically significant” or “positive” results are more likely to be published than results that are “not statistically significant,” “negative,” “null,” “inconclusive,” or otherwise countervailing (Axford et al., 2020; Dwan et al., 2013; Fanelli, 20…”
BackgroundThe empirical march: Making science better at self-correction.
Makel, M. C. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035803 · Cited by 35
“In fact, social scientists appear better at predicting outcomes than researchers from any other field (Fanelli, 2010, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), including those more deeply grounded in theory and with more rigorous or agreed-upon methods (e.g., space science,…”
BackgroundRevisiting the Association between Environmental Tobacco Smoke Exposure and Lung Cancer Risk
Lee, P. N., Fry, J., & Forey, B. (2000) · Indoor and Built Environment · doi:10.1159/000064243 · Cited by 32
Sohn, D. (1996) · Theory & Psychology · doi:10.1177/0959354396062002 · Cited by 32
Probability Pyramiding, Research Error and the Need for Independent Replication
Neher, A. (1967) · doi:10.1007/bf03393713 · Cited by 32
Cerezo, R., Fernández, E., Amieiro, N., Valle, A., Rosário, P. J. S. L. F., & Núñez, J. C. (2018) · Revista de Psicodidáctica (English ed ) · doi:10.1016/j.psicoe.2018.09.001 · Cited by 37
“It was in the middle of the last century when Sterling discussed bout the publication bias and how statistically significant results idáctica, 2019, 24 (1) , 1–8 capture all the attention (Sterling, 1959).”
BackgroundImproving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 40
“And that preference for such results does indeed exist in the form of publication bias, which has been well documented for decades (Atkinson et al., 1982; Franco et al., 2014; Mahoney, 1977; Sterling, 1959).”
BackgroundRose, E. M., Mathew, T., Coss, D. A., Lohr, B., & Omland, K. E. (2018) · Animal Behaviour · doi:10.1016/j.anbehav.2018.09.004 · Cited by 38
A meta-analysis of Libet-style experiments
Braun, M. N., Wessler, J., & Friese, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.018 · Cited by 63
“Publication bias emerges because authors are less likely to submit “failed” studies for publication, and if they do, reviewers and editors are less likely to endorse publication compared to studies that “worked” (e.g., Egger et al., 1997; Fanelli, 2010; Sterling, 1959; Sterling et al., 1995; Tramer et al., 1997).”
Volery, L., Jatavallabhula, D., Scillitani, L., Bertolino, S., & Bacher, S. (2020) · Global Change Biology · doi:10.1111/gcb.15467 · Cited by 38
“We advocate the creation of a public, freely-accessible Global Impact Database in which all impact reports could be deposited (see also Strubbe et al., 2019).”
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 39
Cairo, A. H., Green, J. D., Forsyth, D. R., Behler, A. M. C., & Raldiris, T. L. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220903896 · Cited by 32
2D:4D Suggests a Role of Prenatal Testosterone in Gender Dysphoria
Sadr-Bazzaz, M., Khorashad, B. S., Talaeı, A., Fazeli, N., & Hönekopp, J. (2020) · Archives of Sexual Behavior · doi:10.1007/s10508-020-01630-0 · Cited by 35
“A preference for the publication of statistically significant studies can cause a systematic bias in the published literature (e.g., Sterling, 1959), which is a natural concern for meta-analysis (and any other type of literature review).”
Button, K. S., Chambers, C., Lawrence, N., & Munafo, M. R. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719857659 · Cited by 40
“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results (Franco, Malhotra, & Simonovits, 2014; Sterling, 1959, 1975).”
ResultEffects of Change Interventions
Barends, E., Janssen, B., ten Have, W., & ten Have, S. (2014) · The Journal of Applied Behavioral Science · doi:10.1177/0021886312473152 · Cited by 30
“Since the 1950s, scholars have noted that systematic reviews based only on published studies are subject to the “file drawer problem” (Dickersin, 1990; Egger & Smith, 1998; Rosenthal, 1979; Sterling, 1959).”
Background'True Believers' or Numerical Terrorism at the Nuclear Power Plant
Krämer, W., & Arminger, G. (2010) · Jahrbücher für Nationalökonomie und Statistik · doi:10.1515/jbnst-2011-5-604 · Cited by 20
Paracetamol use and risk of ovarian cancer: a meta-analysis.
Bonovas, S., Filioussi, K., & Sitaras, N. (2006) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2005.02526.x · Cited by 30
Sigelman, L. (1999) · Political Analysis · doi:10.1093/oxfordjournals.pan.a029813 · Cited by 21
“In many other research fields the frequency at whichH0 is rejected is even higher, routinely topping 90% and sometimes even approaching 100% (Sterling 1959; Sterling et al. 1995).”
BackgroundModeling publication bias using weighted distributions in a Bayesian framework
Larose, D. T., & Dey, D. K. (1998) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(97)00039-x · Cited by 35
Estimating the Effect of the File Drawer Problem in Meta-Analysis
Bradley, M. T., & Gupta, R. D. (1997) · Perceptual and Motor Skills · doi:10.2466/pms.1997.85.2.719 · Cited by 30
Hypnosis, psi, and the psychology of anomalous experience
Nadon, R., & Kihlstrom, J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054789 · Cited by 30
Wagner, J. (2021) · International Journal of Social Research Methodology · doi:10.1080/13645579.2021.1922805 · Cited by 29
“For example, Sterling (1959) reviewed four psychology journals published between 1955 and 1956 and reported an unexpectedly high proportion of studies with statistically significant results.”
BackgroundA step-by-step guide on preregistration and effective data sharing for psychopathology research.
Krypotos, A., Klugkist, I. G., Mertens, G., & Engelhard, I. M. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000424 · Cited by 43
Young men and women’s implicit attitudes towards consensually nonmonogamous relationships
Thompson, A. E., Bagley, A., & Moore, E. A. (2018) · Psychology and Sexuality · doi:10.1080/19419899.2018.1435560 · Cited by 37
Psychological science’s replicability crisis and what it means for science in the courtroom.
Chin, J. M. (2014) · Psychology Public Policy and Law · doi:10.1037/law0000012 · Cited by 39
Alpha Values as a Function of Sample Size, Effect Size, and Power: Accuracy over Inference
Bradley, M. T., & Brand, A. (2013) · Psychological Reports · doi:10.2466/03.49.pr0.112.3.835-844 · Cited by 29
“Although NHST has been strongly criticized for decades (Sterling, 1959; Cohen, 1994; Nickerson, 2000), experience shows the progress towards adoption of other approaches such as the construction of confi dence intervals (CIs) for eff ect sizes and Bayesian analysis has been minimal at best.”
MethodReducing Publication Bias Through Trial Registration
Abaid, L., Grimes, D., & Schulz, K. (2007) · Obstetrics and Gynecology · doi:10.1097/01.aog.0000266557.11064.2a · Cited by 29
P‐Hacking , P‐Curves , and the PSM –Performance Relationship: Is There Evidential Value?
Vogel, D., & Homberg, F. (2020) · Public Administration Review · doi:10.1111/puar.13273 · Cited by 33
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 154
Some properties of p-curves, with an application to gradual publication bias.
Ulrich, R., & Miller, J. O. (2017) · Psychological Methods · doi:10.1037/met0000125 · Cited by 36
Metadata
- Authors
- Sterling, T. D.
- Affiliations
- University of Cincinnati (US)
- Corresponding author
- Sterling, Theodore D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1959-03
- Volume
- 54
- Issue
- 285
- Pages
- 30-34
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2062908954
- Cited by
- 964
- References
- 0
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
