The DOI matches its Crossref record.
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract
There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Sterling, 1959
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“These causes of overestimation can dramatically increase Type I error rates in the meta-analysis (Sterling, 1959).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Sterling, 1959
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- 1958, Cancer Research 18
- 1957, Publication Manual,
- Edwards A. L., 1950, Experimental Design in Psychological Research
- Lewis D., 1949, Psychological Bulletin, 46, 433, 10.1037/h0059088 · DOI
- Lubin A., 1957, American Psychologist, 8, 519, 10.1037/h0039746 · DOI
- McNemar, 1955, Psychological Statistics,
- Savage L. J., 1954, The Foundations of Statistics
- Walker H. M., 1947, Elementary Statistical Methods
Cited by
Sorted by citations
Items 401–500 of 888
The Political Determinants of Foreign Direct Investment: A Firm-Level Analysis
Arel‐Bundock, V. (2016) · International Interactions · doi:10.1080/03050629.2016.1185011 · Cited by 25
The predictive validity of peer review: A neglected issue
Bornstein, R. F. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065717 · Cited by 24
Sotola, L. K., & Credé, M. (2022) · European Journal of Social Psychology · doi:10.1002/ejsp.2858 · Cited by 28
Thinking about effect sizes: From the replication crisis to a cumulative psychological science.
Flora, D. (2020) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000218 · Cited by 27
Global positioning system tracking devices can decrease Greater Sage-Grouse survival
Severson, J., Coates, P. S., Prochazka, B., Ricca, M., Casazza, M., & Delehanty, D. (2019) · The Condor · doi:10.1093/condor/duz032 · Cited by 27
“The lack of previously published studies on GPS effects on sage-grouse may stem from a “file drawer effect” (i.e. not publishing nonsignificant results; Sterling 1959, Rosenthal 1979, Barron et al.”
BackgroundMychailyszyn, M. P., & Elson, D. M. (2018) · Children and Youth Services Review · doi:10.1016/j.childyouth.2018.02.011 · Cited by 27
“For instance, the inclusion of non-published studies during the collection period allowed for a more complete sample of studies and took into account the possibility of publication bias (Smith, 1980; Sterling, 1959).”
BackgroundSchlossmacher, I., Junghöfer, M., Straube, T., & Bruchmann, M. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.09.034 · Cited by 27
Publication Bias and Its Implications for Evidence‐Based Clinical Decision Making
Crawford, J. M., Briggs, C. L., & Engeland, C. G. (2010) · Journal of Dental Education · doi:10.1002/j.0022-0337.2010.74.6.tb04904.x · Cited by 25
Meta‐Analysis of Studies with Missing Data
Yuan, Y., & Little, R. J. A. (2009) · Biometrics · doi:10.1111/j.1541-0420.2008.01068.x · Cited by 27
“Concerning publication bias, Sterling (1959) reported that a large proportion of published studies had rejected the null hypotheses, and later studies include Rosenthal (1978, 1979), Dear and Begg (1992), Hedges (1992), Copas (1999), Duval and Tweedie (2000), Baker and Jackson (2006), Henmi, Copas,…”
BackgroundTop ten errors of statistical analysis in observational studies for cancer research
Carmona‐Bayonas, A., Jiménez‐Fonseca, P., Fernández‐Somoano, A., Alvarez-Manceñido, F., Castañón, E., Custodio, A., de la Peña, F. A., Martín‐Payo, R., & Valiente, L. P. (2017) · Clinical & Translational Oncology · doi:10.1007/s12094-017-1817-9 · Cited by 33
Published estimates of group differences in multisensory integration are inflated
Magnotti, J., & Beauchamp, M. (2018) · bioRxiv · doi:10.1371/journal.pone.0202908 · Cited by 26
“preparation and submission phase in which only significant results are included(18-20).”
Backgroundvan der Steen, J. T., van den Bogert, C. A., van Soest‐Poortvliet, M. C., FARSANI, S. F. F., Otten, R. H. J., ter Riet, G., & Bouter, L. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0188247 · Cited by 29
“There are numerous reports that suggest that selective publication is a major problem in the clinical domain, but it is also pervasive in basic and translational research [12, 13] and in the social sciences [14].”
BackgroundEfficacy of Cognitive-Behavioral Therapy in Pediatric Obsessive-Compulsive Disorder: A Meta-Analysis
Wu, Y.-Q., Lang, Z., & Zhang, H. (2016) · Medical Science Monitor · doi:10.12659/msm.895481 · Cited by 26
“Thus, some non-significant studies could have been missed in a meta-analysis of published studies, so the overall effect size could be overestimated [24].”
BackgroundSimonsohn, U. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463399 · Cited by 26
“Virtually all published studies are significant (see, e.g., Fanelli, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), and most studies are underpowered (see, e.”
Statistical Conclusion Validity for Organizational Science Researchers: A Review
Austin, J. T., Boyle, K. A., & Lualhati, J. (1998) · Organizational Research Methods · doi:10.1177/109442819812002 · Cited by 26
Statistical significance testing should be discontinued in mathematics education research
Menon, R. R. (1993) · Mathematics Education Research Journal · doi:10.1007/bf03217248 · Cited by 26
Brand, A., & Bradley, M. T. (2016) · Psychological Reports · doi:10.1177/0033294115625265 · Cited by 25
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292279 · Cited by 24
“The awareness of the existence of this type of selection bias in published literature goes back to at least 1959, when Sterling stated that studies reporting statistically significant findings have a higher probability of being published than studies reporting statistically non-significant findings [13].”
ResultCoretta, S., Casillas, J. V., Roessig, S., Franke, M., Ahn, B., Al‐Hoorie, A. H., Al‐Tamimi, J., Alotaibi, N. E., AlShakhori, M. K., Altmiller, R. M., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231162567 · Cited by 52
“Another possible outcome of analytic flexibility as seen in this study is selective reporting of those tests that yield a desirable outcome (John et al., 2012; Kerr, 1998; Simmons et al., 2011), while null results remain unreported (Rosenthal, 1979; Sterling, 1959).”
BackgroundResultBender, Y., Bräuer, J., & Schweinberger, S. R. (2023) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2023.105857 · Cited by 36
A Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 36
“As a result, null results often end up in the researcher’s file drawer (Sterling, 1959) where they are inaccessible to the research community.”
BackgroundFrom Discovery to Justification: Outline of an Ideal Research Program in Empirical Psychology
Witte, E. H., & Zenker, F. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01847 · Cited by 31
“But this estimate will be unbiased only if we also base it on (often unpublished) statistically non-significant results (Sterling, 1959; Scargle, 2000; Schonemann and Scargle, 2008; Ferguson and Heene, 2012).”
BackgroundInflation von falsch-positiven Befunden in der psychologischen Forschung
Ulrich, R., Erdfelder, E., Deutsch, R., Strauss, B. M., Brüggemann, A., Hannover, B., Tuschen‐Caffier, B., Kirschbaum, C., Blickle, G., Møller, J., et al. (2016) · Psychologische Rundschau · doi:10.1026/0033-3042/a000296 · Cited by 35
Assessment of publication bias for the surgeon scientist
Mahid, S. S., Qadan, M., Hornung, C. A., & Galandiuk, S. (2008) · British journal of surgery · doi:10.1002/bjs.6302 · Cited by 27
Stracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 49
“…avoid duplications of studies but are not often reported and published or not even submitted due to reporting and publication bias, in particular, in social sciences (Fanelli, 2010; Sterling, 1959, Cook&Therrien, 2017; Therrien & Cook, 2018; Franco, Malhotra & Simonovits, 2014; Greenwald, 1975).”
BackgroundPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30
The Replication Crisis in Epidemiology: Snowball, Snow Job, or Winter Solstice?
Lash, T. L., Collin, L. J., & Dyke, M. E. V. (2018) · Current Epidemiology Reports · doi:10.1007/s40471-018-0148-x · Cited by 35
“Publication bias [68, 69], p-hacking [70], HARKing [71], and significance questing [67, 72] are manifestations of this problem.”
A Bayesian “Fill-In” Method for Correcting for Publication Bias in Meta-Analysis
Du, H., Liu, F., & Wang, L.-J. (2017) · Psychological methods · doi:10.1037/met0000164 · Cited by 24
“, over 90%) of significant results in the published literature of social sciences (e.g., Bozarth & Roberts, 1972; Sterling, 1959), which is not compatible with the fact that many psychological studies are underpowered.”
ResultWhat's Wrong with Research Literatures? And how to Make Them Right
Howard, G. S., Hill, T. L., Maxwell, S. E., Baptista, T. M., Farias, M., Coelho, C., Coulter-Kern, M., & Coulter-Kern, R. (2009) · Review of General Psychology · doi:10.1037/a0015319 · Cited by 23
“313) Gigerenzer’s (1993) summary reflects an enormous literature (e.g., Atkinson, Furlong, & Wampold, 1982; Chase & Chase, 1976; Greenwald, 1975; Kerr, Tolliver, & Petree, 1977; Mahoney, 1977; Sterling, 1959) that demonstrates editors’ and reviewers’ preference for significant results (and their bias against nonsignif…”
BackgroundA Beauty Contest of Referee Processes of Economics Journals
Seidl, C., Schmidt, U., & Grösche, P. (2008) · doi:10.1628/186183408785112520 · Cited by 24
The File Drawer Effect and Publication Rates in Menstrual Cycle Research
Sommer, B. A. (1987) · Psychology of Women Quarterly · doi:10.1111/j.1471-6402.1987.tb00786.x · Cited by 24
Großek, A., Großek, K., & Bloch, W. (2023) · Supportive Care in Cancer · doi:10.1007/s00520-023-07773-9 · Cited by 23
“More generally, since studies on publication biases show that it is more likely to publish a study if it demonstrates beneficial effects [36, 37], researchers might feel a constrain to over-interpretation or extending pilot and feasibility studies beyond their defined scope [38].”
BackgroundPharmacotherapy of impulsive aggression: A quality comparison of controlled studies
Felthous, A. R., Lake, S. L., Rundle, B. K., & Stanford, M. S. (2013) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2013.04.017 · Cited by 22
Jackson, D. (2007) · Biometrics · doi:10.1111/j.1541-0420.2006.00663.x · Cited by 23
“There has been an awareness of bias in published results for some time (Sterling, 1959), including medical data (Begg and Berlin, 1988; Easterbrook et al., 1991).”
BackgroundImaging and Diagnostic Testing
Ekery, D., Davidoff, R., Orlandi, Q. G., Apstein, C. S., Hesselvik, J. F., Shemin, R. J., & Shapira, O. M. (2003) · American Heart Journal · doi:10.1016/s0002-8703(02)94786-1 · Cited by 28
Methodological versus empirical literature: Two views on casual acceptance of the null hypothesis.
Harcum, E. R. (1990) · American Psychologist · doi:10.1037/0003-066x.45.3.404 · Cited by 23
der Cruyssen, I. V., Ben‐Shakhar, G., Pertzov, Y., Guy, N., Cabooter, Q., Gunschera, L. J., & Verschuère, B. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02221-2 · Cited by 23
“It is argued that this is caused by exaggerated effect size estimates in the existing literature due to a combination of publication bias and questionable research practices (e.g., Simmons et al., 2011; Sterling, 1959).”
BackgroundHeterogeneity estimates in a biased world
Hönekopp, J., & Linden, A. H. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262809 · Cited by 25
“Publication bias arises when studies with statistically non-significant results have a reduced chance of being published [20].”
BackgroundMcClain, M. B., Callan, G. L., Harris, B., Floyd, R. G., Haverkamp, C. R., Golson, M. E., Longhurst, D. N., & Benallie, K. J. (2021) · Journal of School Psychology · doi:10.1016/j.jsp.2020.11.002 · Cited by 24
How often does random assignment fail? Estimates and recommendations
Goldberg, M. H. (2019) · Journal of Environmental Psychology · doi:10.31219/osf.io/s2j4r · Cited by 22
Mychailyszyn, M. P. (2017) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000101 · Cited by 29
“The rigorous standards of the peer-review process have led to the speculation that studies which have made it to print are more likely to have obtained statistically significant findings as compared with those that were never published (Sterling, 1959).”
Resultp-Values: The Insight to Modern Statistical Inference
Fraser, D. (2017) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-060116-054139 · Cited by 22
Yu, D.-J., Li, Y., Hu, J.-D., Yang, X.-B., Yang, J.-Y., & Shen, H.-B. (2015) · IEEE/ACM Transactions on Computational Biology & Bioinformatics · doi:10.1109/tcbb.2014.2359451 · Cited by 22
When nonreliability of reviews indicates solid science
Eckberg, D. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065791 · Cited by 21
Smith, S. R., Kroon, J., Schwarzer, R., & Hamilton, K. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113322 · Cited by 27
Is Profit Sharing Productive? A Meta‐Regression Analysis
Doucouliagos, H., Laroche, P., Kruse, D., & Stanley, T. (2020) · British Journal of Industrial Relations · doi:10.1111/bjir.12483 · Cited by 21
Schreiber, J. B. (2020) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2019.05.023 · Cited by 21
Elliot, A. J., & Maier, M. (2013) · Journal of Experimental Psychology General · doi:10.1037/a0029592 · Cited by 22
Registering CIHR-funded randomized controlled trials: a global public good
Moher, D. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041299 · Cited by 16
The logic and role of meta-analysis in clinical research
Sandoz, M. (1993) · Statistical Methods in Medical Research · doi:10.1177/096228029300200203 · Cited by 21
Girardi, P., Vesely, A., Lakens, D., Altoè, G., Pastore, M., Calcagnì, A., & Finos, L. (2024) · Psychometrika · doi:10.1007/s11336-024-09973-6 · Cited by 20
“When researchers select and report the results of a subset of all possible analyses that produce significant results (Sterling, 1959; Greenwald, 1975; Simmons et al., 2011; Brodeur et al., 2016), they dramatically increase the actual false-positive rates despite their nominal endorsement of a low type I error rate (e.”
BackgroundWhy ex post peer review encourages high-risk research while ex ante review discourages it
Gross, K., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2111615118 · Cited by 25
“The momentum towards ex ante journal review is intended to remedy the selective reporting, datadredging, p-hacking, and other questionable research practices (QRPs) that conventional, outcome-based (ex post) review encourages [41, 42].”
BackgroundModelling publication bias and p ‐hacking
Moss, J., & Bin, R. D. (2021) · Biometrics · doi:10.1111/biom.13560 · Cited by 21
“Publication bias (Sterling, 1959) and p -hacking (Simmons et al. , 2011) are the most common phenomena that violate these assumptions.”
BackgroundFidler, F. M. (2018) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/7xdpq · Cited by 48
“Sterling (1959) surveyed articles in four leading psychology journals(10) published in 1955 and 1956.”
BackgroundInfluentialFact or fiction: reducing the proportion and impact of false positives
Stahl, D. R., & Pickles, A. R. (2017) · Psychological Medicine · doi:10.1017/s003329171700294x · Cited by 23
The Impact of Human Intuition in Psychology
Goddard, M. J. (2009) · Review of General Psychology · doi:10.1037/a0015104 · Cited by 20
Reducing publication bias of prospective clinical trials through trial registration
Abaid, L. N., Grimes, D. A., & Schulz, K. F. (2007) · Contraception · doi:10.1016/j.contraception.2007.06.013 · Cited by 18
The Evaluation of Epidemiologic Evidence for Policy-making
Szklo, M. (2001) · American Journal of Epidemiology · doi:10.1093/aje/154.12.s13 · Cited by 18
“Concern about publication bias is not new (22), but has been underscored more recently by the increasing popularity of meta-analysis of published studies as a way to summarize results for policy purposes (23).”
BackgroundShaffer, J. W., Duszynski, K. R., & Thomas, C. B. (1982) · Journal of Behavioral Medicine · doi:10.1007/bf00844805 · Cited by 43
How Do You Test a Test?: A Multifaceted Examination of Significance Tests
Ferro, N., & Sanderson, M. (2022) · Web Search and Data Mining · doi:10.1145/3488560.3498406 · Cited by 19
“It has long been recognized that a side effect of a researcher obtaining significance in an experimental result is that the result appears to have a greater chance of publication [39], the effect is known as publication bias .”
BackgroundMayo‐Wilson, E., Grant, S., & Supplee, L. H. (2021) · Prevention Science · doi:10.1007/s11121-021-01284-x · Cited by 25
“Greater transparency in the design, execution, and analysis of intervention evaluations is paramount to the translation of prevention research into evidence-based policy (Supplee & Meyer, 2015).”
BackgroundLinder, C., & Farahbakhsh, S. (2020) · Business Ethics Quarterly · doi:10.1017/beq.2019.52 · Cited by 24
Ego Depletion May Disappear by 2020
Vadillo, M. (2019) · Social Psychology · doi:10.31234/osf.io/esjhp · Cited by 19
O. Collaboration (2018) · Implementing Reproducible Research · doi:10.1201/9781315373461-11 · Cited by 19
“Layered on top of legitimate epistemological considerations are cultural forces that favor significant (Fanelli, 2010, 2012; Greenwald, 1975; Sterling, 1959) and consistent (Giner-Sorolla, 2012) results over inconsistent or ambiguous results.”
BackgroundReasoning from Paradigms and Negative Evidence
Macagno, F., & Walton, D. (2011) · Pragmatics & Cognition · doi:10.1075/pc.19.1.04mac · Cited by 19
Publication bias and journals as policemen
DeMaria, A. N. (2004) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2004.09.018 · Cited by 17
Statistical Significance: Balancing Evidence Against Doubt
Hall, P., & Selinger, B. (1986) · doi:10.1111/j.1467-842x.1986.tb00708.x · Cited by 19
Adjuvant Therapy of Colorectal Cancer: Why We Still Don't Know
Buyse, M. (1988) · JAMA · doi:10.1001/jama.1988.03720240033031 · Cited by 18
Publication bias and genuine effects: the case of Granger causality between tourism and income
Fonseca, N., & Sánchez-Rivero, M. (2020) · Current Issues in Tourism · doi:10.1080/13683500.2019.1585419 · Cited by 18
“In this paper, we present a meta-analysis of the results concerned to the tests of Granger causality between tourism and income (Brida, Cortes-Jimenez, & Pulina, 2016; Pablo-Romero & Molina, 2013), taking into account the problem of publication bias (Humphreys, Reigel, & Epstein, 1955; Sterling, 1959), in the light of…”
BackgroundMethodA Path to Greater Credibility: Large-Scale Collaborative Education Research
Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019) · AERA Open · doi:10.1177/2332858419891963 · Cited by 27
“For example, there is a long history of publication bias favoring statistically significant results (Polanin, Tanner-Smith, & Hennessy, 2016; Sterling, 1959), which is itself extremely troublesome.”
BackgroundThe Publication Incidence of Replications and Critical Commentary in Economics
Hubbard, R., & Vetter, D. E. (1992) · doi:10.1177/056943459203600106 · Cited by 18
Reflections on the peer review process
Marsh, H. W., & Ball, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065912 · Cited by 18
Confusion between reviewer reliability and wise editorial and funding decisions
Kiesler, C. A. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065869 · Cited by 16
Sterling, T. D. (1971) · Science · doi:10.1126/science.174.4016.1358 · Cited by 18
Self-citationEvidence of citation bias in the pesticide ecotoxicology literature
Hanson, M. L., Deeth, L., & Prosser, R. S. (2018) · Ecotoxicology · doi:10.1007/s10646-018-1918-4 · Cited by 17
Publication of Nonsignificant Results: A Survey of Psychologists' Opinions
Reysen, S. (2006) · Psychological Reports · doi:10.2466/pr0.98.1.169-175 · Cited by 17
A toolbox to evaluate the trustworthiness of published findings
Adler, S. J., Röseler, L., & Schöniger, M. K. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114189 · Cited by 21
Preregistration in experimental linguistics: applications, challenges, and limitations
Roettger, T. B. (2021) · Linguistics · doi:10.1515/ling-2019-0048 · Cited by 51
“Confirmatory analyses have a superior status within the academic incentive system, determining the way funding agencies assess proposals, and shaping how researchers frame their papers (Sterling 1959).”
BackgroundStatistical Tests and Substantive Significance
Gold, D. (2017) · doi:10.4324/9781315134918-25 · Cited by 16
The Reinhart-Rogoff controversy as an instance of the ‘emerging contrary result’ phenomenon
Maziarz, M. (2017) · doi:10.1080/1350178x.2017.1302598 · Cited by 16
“Publication bias, the term coined by Sterling (1959), is usually defined as preference expressed by journal editors for statistically significant findings.”
BackgroundFour Classic Public Goods Experiments: A Replication Study
Eckel, C. C., Harwell, H., & Castillo, G. (2015) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0193-230620150000018001 · Cited by 16
Negative Results in European Psychology Journals
Vasilev, M. R. (2013) · Europe’s Journal of Psychology · doi:10.5964/ejop.v9i4.590 · Cited by 12
“The main reason for this is that the studies that have investigated the problem are few and far in between (Bozarth & Roberts, 1972; Spence & Blanchard, 2001; Sterling, 1959; Sterling, Rosenbaum, &Weinkam, 1995).”
BackgroundPublishing integrative landscape research: analysis of editorial policies of peer-reviewed journals
Tress, G., Tress, B., & Fry, G. (2006) · Environmental Science & Policy · doi:10.1016/j.envsci.2006.03.004 · Cited by 16
Diagnosing Estimate Distortion Due to Significance Testing in Literature on Detection of Deception
Bradley, M. T., & Stoica, G. (2004) · Perceptual and Motor Skills · doi:10.2466/pms.98.3.827-839 · Cited by 16
Reflections from the peer review mirror
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066024 · Cited by 16
Inference from Empirical Research.
Burgstahler, D. (1987) · Accounting Review · doi:10.2308/tar-4499177 · Cited by 16
The Statistical Power of a Decade of Social Work Education Research
Orme, J. G. E., & Tolman, R. (1986) · Social Service Review · doi:10.1086/644403 · Cited by 16
Pitfalls of significance testing and $p$-value variability: An econometrics perspective
Hirschauer, N., Grüner, S., Mußhoff, O., & Becker, C. (2018) · Statistics Surveys · doi:10.1214/18-ss122 · Cited by 21
“Starting with Sterling (1959) as an early precursor, researchers have increasingly realized this bias in recent years.”
BackgroundPublication rates of editorial board members in oral health journals
Rõsing, C. K., Junges, R., & Haas, A. N. (2014) · Brazilian Oral Research · doi:10.1590/1807-3107bor-2014.vol28.0042 · Cited by 19
Park, S., & Chan, K. K. H. (2025) · Review of Educational Research · doi:10.3102/00346543251394404 · Cited by 15
Reckoning With Our Crisis: An Agenda for the Field of Social and Personality Psychology
Schiavone, S. R., & Vazire, S. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221101060 · Cited by 23
“…results can help reveal a potentially troublingly rate of statistically significant results in the published literature in psychology—or, at the least, spur conversation about what the optimal proportion should be (Fanelli, 2010; Scheel et al., 2021; Sterling, 1959; Sterling et al., 1995).”
ResultDissimulate or disseminate? A survey on the fate of negative results
Herbet, M., Léonard, J., Santangelo, M., & Albaret, L. (2022) · Learned Publishing · doi:10.1002/leap.1438 · Cited by 17
Quality Control for Scientific Research: Addressing Reproducibility, Responsiveness, and Relevance
Hubbard, D. W., & Carriquiry, A. L. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543138 · Cited by 20
“Sterling (1959), Rosenthal (1979), and Sackett (1979) discussed the effect of publication bias on reproducibility in various scientific disciplines over 50 years ago.”
BackgroundEffect Size Estimation Fromt-Statistics in the Presence of Publication Bias
Ulrich, R., Miller, J. O., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 20
“Also, critics have long suspected that journal editors prefer to publish positive results and that therefore researchers are reluctant to submit negative results (Franco, Malhotra, & Simonovits, 2014; Smart, 1964; Sterling, 1959).”
BackgroundResultEnvironmental tobacco smoke exposure and lung cancer: A systematic review
Lee, P. N., Fry, J., Forey, B., Hamling, J. S., & Thornton, A. (2016) · doi:10.13105/wjma.v4.i2.10 · Cited by 15
22. Power Statistics for Meta-analysis: Tests for Mean Effects and Homogeneity
Lajeunesse, M. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-024 · Cited by 15
Riniolo, T. C. (1997) · Teaching of Psychology · doi:10.1207/s15328023top2404_15 · Cited by 15
“Although formal studies have shown that the majority of articles published in journals are filled with positive results (Bozarth & Roberts, 1972; Sterling, 1959), introductory rtsearch methods textbooks (e.g., Graziano & Raulin, 1989; Ray, 1993) often fail to alert students that published research…”
BackgroundSchoonen, R. (1991) · Brain and Language · doi:10.1016/0093-934x(91)90166-x · Cited by 15
“A lot of research output might thus remain unpublished (Sterling, 1970; Lykken, 1970).”
BackgroundChapter I: Hypothesis Testing in Relation to Statistical Methodology
Clark, C. A. (1963) · Review of Educational Research · doi:10.3102/00346543033005455 · Cited by 15
Are Temporary Jobs Stepping Stones or Dead Ends? A Meta-Analytical Review of the LiteraturePreprint
Filomena, M., & Picchio, M. (2021) · Social Science Research Network · doi:10.2139/ssrn.3851042 · Cited by 15
Metadata
- Authors
- Sterling, T. D.
- Affiliations
- University of Cincinnati (US)
- Corresponding author
- Sterling, Theodore D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1959-03
- Volume
- 54
- Issue
- 285
- Pages
- 30-34
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2062908954
- Cited by
- 964
- References
- 0
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
