The DOI matches its Crossref record.
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract
There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Sterling, 1959
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“These causes of overestimation can dramatically increase Type I error rates in the meta-analysis (Sterling, 1959).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Sterling, 1959
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- 1958, Cancer Research 18
- 1957, Publication Manual,
- Edwards A. L., 1950, Experimental Design in Psychological Research
- Lewis D., 1949, Psychological Bulletin, 46, 433, 10.1037/h0059088 · DOI
- Lubin A., 1957, American Psychologist, 8, 519, 10.1037/h0039746 · DOI
- McNemar, 1955, Psychological Statistics,
- Savage L. J., 1954, The Foundations of Statistics
- Walker H. M., 1947, Elementary Statistical Methods
Cited by
Sorted by citations
Items 201–300 of 888
Small-Study Effects in Meta-Analysis
Schwarzer, G., Carpenter, J. R., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0_5 · Cited by 114
Modelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103
#EEGManyLabs: Investigating the Replicability of Influential EEG ExperimentsPreprint
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/528nr · Cited by 26
“These findings are consistent with a high degree of publication bias (Francis, 2012; Ioannidis, 2005; Ku¨hberger et al., 2014; Sterling, 1959).”
Gentile, A., Boca, S., & Giammusso, I. (2018) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2018.07.013 · Cited by 142
“Publication bias (also known as ‘file-drawer problem’) is the main weakness of a meta-analysis and occurs because many journal editors are often reluctant to publish ‘non-significant’ results, thus leaving the so-called null results in the file drawer (Ioannidis, 2005; Rosenthal, 1979; Sterling, 1959) This practice ca…”
BackgroundThe frequency of excess success for articles in Psychological Science
Francis, G. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0601-x · Cited by 130
“Probabilistic reasoning It is widely recognized that a bias exists across articles in the field of psychology (Fanelli, 2010; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Methodological factors in mere exposure research.
Stang, D. (1974) · Psychological Bulletin · doi:10.1037/h0037419 · Cited by 98
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 101
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 100
“Findings that nearly all of the research reports published in psychology report significant, positive results (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995) suggest research supporting the null hypothesis is seldom published.”
ResultThe role of the funnel plot in detecting publication and related biases in meta-analysis
Liu, J. L. (2011) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6400831 · Cited by 118
Multiple comparisons and association selection in general epidemiology.
Greenland, S. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn064 · Cited by 95
A comparative meta-analysis of Rorschach and MMPI validity.
Hiller, J. B., Rosenthal, R., Bornstein, R., Berry, D. T. R., & Brunell-Neuleib, S. (1999) · Psychological Assessment · doi:10.1037/1040-3590.11.3.278 · Cited by 93
“A question often asked in meta-analysis is whether the studies included may have a bias toward larger effect sizes, because studies with significant findings are more likely to be published (Greenwald, 1975; Sterling, 1959) and are therefore easier to retrieve (Rosenthal, 1979).”
BackgroundComment: Assumptions and Procedures in the File Drawer Problem
Rosenthal, R., & Rubin, D. (1988) · Statistical Science · doi:10.1214/ss/1177013014 · Cited by 93
Taylor, P., Reynolds, R., Calhoun, V., Gonzalez-Castillo, J., Handwerker, D. A., Bandettini, P., Mejia, A. F., & Chen, G. (2022) · bioRxiv · doi:10.1101/2022.10.26.513929 · Cited by 91
“This feature is related to the classic "publication bias" problem, wherein findings with weak or negative evidence do not get reported, which biases meta-analyses (Sterling, 1959; Ioannidis, 2008; Jennings and Van Horn, 2012).”
Schwarzer, G., Antes, G., & Schumacher, M. C. (2002) · Statistics in Medicine · doi:10.1002/sim.1224 · Cited by 81
Treatment of posttraumatic stress disorder in U.S. combat Veterans: a meta-analytic review
Goodson, J., Helstrom, A., Halpern, J. M., Ferenschak, M. P., Gillihan, S. J., & Powers, M. B. (2011) · Psychological Reports · doi:10.2466/02.09.15.16.pr0.109.5.573-599 · Cited by 90
Over-optimism in bioinformatics research
Boulesteix, A. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp648 · Cited by 85
Bak, T. H. (2016) · Linguistic Approaches to Bilingualism · doi:10.1075/lab.16002.bak · Cited by 114
Klassen, T., Wiebe, N., Russell, K., Stevens, K., Hartling, L., Craig, W., & Moher, D. (2002) · Archives of Pediatrics & Adolescent Medicine · doi:10.1001/archpedi.156.5.474 · Cited by 88
Parapsychology: Science of the anomalous or search for the soul?
Alcock, J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054467 · Cited by 88
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
“bias favoring positive results together produce a literature with upwardly biased effect sizes (14, 16, 33, 34).”
BackgroundSignificance testing in empirical finance: a critical review and assessment
Kim, J. H., & Ji, P. I. (2015) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2015.08.006 · Cited by 86
Replication in criminology: A necessary practice
McNeeley, S., & Warner, J. J. (2015) · European Journal of Criminology · doi:10.1177/1477370815578197 · Cited by 93
“In a review of psychology journals, Sterling (1959) found that, out of 362 research articles published in a one-year period, none were replications of earlier work.”
BackgroundDo higher public debt levels reduce economic growth?
Heimberger, P. (2022) · Journal of Economic Surveys · doi:10.1111/joes.12536 · Cited by 107
Chapter 11: Issues in Meta-Analysis
Hedges, L. (1986) · Review of Research in Education · doi:10.3102/0091732x013001353 · Cited by 85
“There is considerable empirical evidence that the published literature contains fewer statistically insignificant results than would be expected from the complete collection of all studies actually con ducted (Bozarth & Roberts, 1972; Hedges, 1984b; Sterling, 1959).”
Few systematic reviews exist documenting the extent of bias: a systematic review
Tricco, A. C., Tetzlaff, J., Sampson, M., Fergusson, D. A., Cogo, E., Horsley, T., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.017 · Cited by 82
Mega‐analysis methods in ENIGMA : The experience of the generalized anxiety disorder working group
Zugman, A., Harrewijn, A., Cardinale, E. M., Zwiebel, H., Freitag, G. F., Werwath, K. E., Bas‐Hoogendam, J. M., Groenewold, N. A., Aghajani, M., Hilbert, K., et al. (2020) · Human Brain Mapping · doi:10.1002/hbm.25096 · Cited by 83
“For example, preregistration reduces problems that follow when negative results remain unreported (Rosenthal, 1979; Sterling, 1959), reduces the chances of selective reporting (Macleod et al., 2014) and maximizes transparency in analytic approaches, thereby facilitating replication (Simmons,…”
BackgroundA Meta-Analysis on the Effectiveness of Bilingual Programs in Europe
Reljić, G., Ferring, D., & Martin, R. (2014) · Review of Educational Research · doi:10.3102/0034654314548514 · Cited by 105
“Moreover, the studies that have been published are more likely to have achieved statistical significance than those unpublished studies still in the file drawers as was noted by Sterling (1959) more than 50 years ago.”
Where Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81
“Multiple fields of science have a very poor record of publishing negative and null results (Sterling et al., 1995; Fanelli, 2010, 2012; Greenwald, 1975; Hubbard & Armstrong, 1992; Franco et al., 2014; Sterling, 1959; Sterling, Rosenbaum, & Wenkam, 1995), suggesting that the published literature is biased and incomplet…”
BackgroundResultNull hypothesis disrespect in neuropsychology: dangers of alpha and beta errors.
Soper, H., Cicchetti, D. V., Satz, P., Light, R., & Orsini, D. L. (1988) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/01688638808408240 · Cited by 81
Magic: A Theoretical Reassessment [and Comments and Replies]
Winkelman, M., Ankenbrandt, K. W., Bharati, A., Bourguignon, E., de Rios, M. D., Dundes, A., Eisenbud, J., Goodman, F., Hallpike, C. R., Hultkrantz, Ā., et al. (1982) · Current Anthropology · doi:10.1086/202778 · Cited by 81
Rost, K., & Ehrmann, T. (2015) · Business & Society · doi:10.1177/0007650315572858 · Cited by 98
“Medical researchers have long acknowledged the importance and seriousness of publication selection (Begg & Berlin, 1988; Egger, Smith, Schneider, & Minder, 1997; Sterling, 1959).”
BackgroundLemmon, V., Ferguson, A. R., Popovich, P. G., Xu, X., Snow, D. M., Igarashi, M., Beattie, C. E., & Bixby, J. L. (2014) · Journal of Neurotrauma · doi:10.1089/neu.2014.3400 · Cited by 79
Macro Aid Effectiveness Research: A Guide for the Perplexed
Roodman, G. D. (2007) · doi:10.2139/ssrn.1101461 · Cited by 80
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79
“Significance Testing (NHST) to evaluate their hypotheses (Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study
Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016) · Comparative Political Studies · doi:10.1177/0010414016655539 · Cited by 80
“…work, publication bias is most pronounced when publication decisions are based on the realized outcomes of a study—typically statistical significance of a result—rather than the merits of the approach and design (Dickersin, 1990; Humphreys, de la Sierra, & van der Windt, 2013; Sterling, 1959).”
BackgroundAssessing publication bias in genetic association studies: evidence from a recent meta-analysis
Munafo, M. R., Clark, T. G., & Flint, J. (2004) · Psychiatry Research · doi:10.1016/j.psychres.2004.06.011 · Cited by 168
“This concern is not new and was raised almost 50 years ago in relation to psychiatric and psychological research (Sterling, 1959).”
BackgroundEstimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76
“First, often non-significant results are not available because journals selectively publish significant results (Rosenthal, 1979; Sterling, 1959; Sterling et al., 1995).”
BackgroundA Case for Registering Studies of Political Outcomes: An Application in the 2010 House Elections
Monogan, J. E. (2013) · Political Analysis · doi:10.1093/pan/mps022 · Cited by 76
“Gill also cites Sterling (1959), who made the case that the unpublished results could mean that researchers will replicate efforts until a Type I error is committed and the study is published.”
BackgroundData management and sharing in neuroimaging: Practices and perceptions of MRI researchersPreprint
Borghi, J., & Gulick, A. E. V. (2018) · bioRxiv · doi:10.1101/266627 · Cited by 75
“Issues such as publication bias 42 , 43 and low statistical power 44 , 45 have been discussed in the behavioral and biomedical sciences for decades and data stakeholders including scholarly publishers and federal funding agencies have instituted a range of reproducibility-related policies…”
BackgroundSponsored symposia on environmental tobacco smoke.
Bero, L., Galbraith, A. A., & Rennie, D. (1994) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1994.03510320052029 · Cited by 75
Trait resilience and mental health in older adults: A meta-analytic review.
Färber, F., & Rosendahl, J. (2020) · Personality and Mental Health · doi:10.1002/pmh.1490 · Cited by 73
Significance Testing in Empirical Finance: A Critical Review and Assessment
Kim, J. H., & Ji, P. (2015) · doi:10.2139/ssrn.2410049 · Cited by 73
“This means the studies with statistically significant results have higher chance of publication than those not; and that many new and important findings may be severely disadvantaged if their results are statistically insignificant (see Sterling, 1959; and Kendall, 1970; p.89).”
Researcher Incentives and Empirical MethodsPreprint
Glaeser, E. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.934557 · Cited by 73
“Denton (1985) follows Sterling (1959) and makes it clear that exactly the same statistical problems occur with data mining and publication bias.”
BackgroundVaitkus, P., & Brar, C. (2007) · American Heart Journal · doi:10.1016/j.ahj.2006.09.014 · Cited by 72
Lecoutre, B., Lecoutre, M., & Poitevineau, J. (2001) · International Statistical Review · doi:10.1111/j.1751-5823.2001.tb00466.x · Cited by 56
“Excellence R Us”: university research and the fetishisation of excellence
Moore, S., Neylon, C., Eve, M. P., O’Donnell, D. P., & Pattinson, D. (2017) · Humanities and Social Sciences Communications · doi:10.1057/palcomms.2016.105 · Cited by 292
“…this need to perform “excellence” is can be illustrated by the wellknown bias towards positive results in scientific publication (for example, Dickersin et al., 1987, 2005; Sterling, 1959; Kennedy, 2004; Young and Bang, 2004; Bertamini and Munafò, 2012; Rothstein, 2014; Psych Filedrawer, 2016).”
BackgroundThe significance filter, the winner's curse and the need to shrink
van Zwet, E. W., & Cator, E. (2021) · Statistica Neerlandica · doi:10.1111/stan.12241 · Cited by 84
“See Hedges (1984), Hedges and Vevea (2005), Lane and Dunlap (1978), Sterling (1959), Sterling, Rosenbaum, and Weinkam (1995) and in particular the monograph Rothstein, Sutton, and Borenstein (2005) for the many issues related to publication bias.”
BackgroundNull Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 78
“Although we did not identify any systematic review of negative findings in the education literature, reviews indicate the predominance of positive findings in the related field of psychology is virtually unchanged since the 1950s (Spence & Blanchard, 2001; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995; Vasilev,…”
BackgroundFritz, A., Scherndl, T., & Kühberger, A. (2012) · Theory & Psychology · doi:10.1177/0959354312436870 · Cited by 77
Are null results becoming an endangered species in marketing?
Hubbard, R., & Armstrong, J. (1992) · Marketing Letters · doi:10.1007/bf00993992 · Cited by 71
“For instance, Sterling's (1959) investigation of papers published in four leading psychology journals during 1955 found that only 2.7% of those using statistical significance tests failed to reject the null hypothesis.”
BackgroundInfluentialThe effect of publication bias on the Q test and assessment of heterogeneity.
Augusteijn, H. E. M., van Aert, R. C. M., & van Assen, M. A. L. M. (2018) · Psychological Methods · doi:10.1037/met0000197 · Cited by 77
Publication bias and public health policy on environmental tobacco smoke.
Bero, L., Glantz, S., & Rennie, D. (1994) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1994.03520020059016 · Cited by 68
The anomaly called psi: Recent research and criticism
Rao, K., & Palmer, J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054455 · Cited by 68
The Role of p -Values in Judging the Strength of Evidence and Realistic Replication Expectations
Gibson, E. W. (2020) · Statistics in Biopharmaceutical Research · doi:10.1080/19466315.2020.1724560 · Cited by 68
Comparing Japan and the United States on individualism/collectivism: A follow‐up review
Takano, Y., & Osaka, E. (2018) · Asian Journal Of Social Psychology · doi:10.1111/ajsp.12322 · Cited by 68
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G., & Moher, D. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-79 · Cited by 66
“Publication bias initially described as the file drawer problem as a bias concept in primary studies was early to emerge and has long been suspected in the social sciences [7].”
BackgroundPublication Bias in Two Political Behavior Literatures
Gerber, A. S., Malhotra, N., Dowling, C. M., & Doherty, D. (2010) · American Politics Research · doi:10.1177/1532673x09350979 · Cited by 67
“A number of studies have identified publication bias in the fields of psychology (e.g., Coursol & Wagner, 1986; Greenwald, 1975; Sterling, 1959), public health and medicine (e.g., Gotzsche, 2006), economics (e.g., Ashenfelter, Harmon, & Oosterbeek, 1999; Card & Krueger, 1995; De Long & Lang, 1992;…”
BackgroundQuestionable science and reproducibility in electrical brain stimulation research
Héroux, M. E., Loo, C. K. A., Taylor, J. L., & Gandevia, S. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175635 · Cited by 78
Meta-analysis of Comparative Therapy Outcome Research: a Critical Appraisal
Shapiro, D. A., & Shapiro, D. (1982) · Behavioural Psychotherapy · doi:10.1017/s0141347300006285 · Cited by 66
“This concerns the above-mentioned bias against the null hypothesis arising from the lower probability that negative results will be written up and published (Bakan, 1966; Greenwald, 1975; Smart, 1964; Sterling, 1959).”
BackgroundLikelihood and Bayesian Inference from Selectively Reported Data
Dawid, A., & Dickey, J. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10479968 · Cited by 66
The Consequences of Journal Editorial Policies and a Suggestion for Revision
Feige, E. (1975) · Journal of Political Economy · doi:10.1086/260400 · Cited by 66
Split-Sample Strategies for Avoiding False Discoveries
Anderson, M. L., & Magruder, J. R. (2017) · National Bureau of Economic Research · doi:10.3386/w23544 · Cited by 65
“This solution naturally opens the door to false discoveries, and limiting these false discoveries is a critical issue in most empirical disciplines (Sterling 1959).”
BackgroundThe implications of publication bias for meta‐analysis' other parameter
Jackson, D. (2006) · Statistics in Medicine · doi:10.1002/sim.2293 · Cited by 65
Honorton, C. (1975) · Journal of Communication · doi:10.1111/j.1460-2466.1975.tb00560.x · Cited by 65
Z-Curve.2.0: Estimating Replication Rates and Discovery Rates
Bartoš, F., & Schimmack, U. (2020) · Meta-Psychology · doi:10.31234/osf.io/urgtn · Cited by 64
“For a number of reasons, most published studies are selected because they reported a theoretically interesting result that was statistically significant (Rosenthal & Gaito, 1964; Scheel, Schijen, & Lakens, 2020; Sterling, 1959; Sterling et al., 1995).”
ResultThe empirical replicability of task-based fMRI as a function of sample size
Bossier, H., Roels, S. P., Seurinck, R., Banaschewski, T., Barker, G. J., Bokde, A. L. W., Quinlan, E. B., Desrivières, S., Flor, H., Grigis, A., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116601 · Cited by 79
Researcher degrees of freedom in phonetic research
Roettger, T. B. (2019) · Laboratory Phonology Journal of the Association for Laboratory Phonology · doi:10.5334/labphon.147 · Cited by 77
“, the tendency to publish positive results more often than null results (Franco, Malhotra, & Simonovits, 2014; Sterling, 1959), as the number of failed attempts to reject a hypothesis can be tracked transparently (if the studies were conducted).”
ResultThe quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 84
A Review of Evidence for the Claim that Children are Sleeping Less than in the Past
Matricciani, L., Olds, T., & Williams, M. (2011) · SLEEP · doi:10.1093/sleep/34.5.651 · Cited by 62
Why most psychological research findings are not even wrongPreprint
Scheel, A. M. (2021) · Infant and Child Development · doi:10.31234/osf.io/8w2sd · Cited by 61
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 73
“There is evidence that the rate of positive findings has stayed approximately the same from the 1950’s (97.3% in psychology; Sterling, 1959) until the 1990s (95.”
BackgroundInfluentialThe experiment as the unit for computing rates of error.
Ryan, T. A. (1962) · Psychological bulletin · doi:10.1037/h0040562 · Cited by 60
Bartos̆, F., Maier, M., Wagenmakers, E., Nippold, F., Doucouliagos, H., Ioannidis, J. P. A., Otte, W. M., Sladekova, M., Deresssa, T. K., Bruns, S. B., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1703 · Cited by 76
Mathur, M. B., & VanderWeele, T. J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 92
A Call for Openness in Research Reporting: How to Turn Covert Practices Into Helpful Tools
Schwab, A., & Starbuck, W. H. (2016) · Academy of Management Learning and Education · doi:10.5465/amle.2016.0039 · Cited by 80
“Decades of prior publications have discussed different aspects and implications of these practices (e.g., Greenwald, 1975; Rosenthal, 1979; Sterling, 1959).”
BackgroundReferential precedents in spoken language comprehension: A review and meta-analysis
Kronmüller, E., & Barr, D. (2015) · Journal of Memory and Language · doi:10.1016/j.jml.2015.03.008 · Cited by 59
“One of the perennial problems with meta-analysis is publication bias and the well-known “file drawer” problem (Rosenthal, 1979; Sterling, 1959), such that the set of published studies is not representative of the underlying effects.”
REPLICATION IN EXPERIMENTAL COMMUNICATION RESEARCH: AN ANALYSIS
Kelly, C. W., Chase, L. J., & Tucker, R. K. (1979) · Human Communication Research · doi:10.1111/j.1468-2958.1979.tb00646.x · Cited by 59
What happened to Popperian falsification? Publishing neutral and negative findings
van Witteloostuijn, A. C. (2016) · Cross Cultural & Strategic Management · doi:10.1108/ccsm-03-2016-0084 · Cited by 63
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 84
“…of a bias against negative results comes from studies that find that the vast majority of results in the scientific literature are positive (Dickersin et al., 1987; Sterling, 1959), particularly in psychology (Fanelli, 2010), despite the common use of underpowered designs (Bakker et al., 2012).”
ResultUnderstanding Statistical Significance
Hayat, M. J. (2010) · Nursing Research · doi:10.1097/nnr.0b013e3181dbb2cc · Cited by 57
Publication bias and the integrity of psychiatry research
Gilbody, S. M., & Song, F. (2000) · Psychological Medicine · doi:10.1017/s0033291700001732 · Cited by 64
“It has long been recognized that research with ‘negative’ or ‘uninteresting’ results is less likely to be published (Sterling, 1959), due to the failure of researchers and sponsors to submit negative research and the failure of editors to publish if it is submitted (Dickersin et al.”
BackgroundFagley, N. (1985) · Journal of Counseling Psychology · doi:10.1037/0022-0167.32.3.391 · Cited by 57
Nuijten, M. B., & Polanin, J. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1408 · Cited by 56
A meta-psychological perspective on the decade of replication failures in social psychology.
Schimmack, U. (2020) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000246 · Cited by 56
Statistical Significance with Comments by Editors of Marketing Journals
Hubbard, R., & Ryan, P. A. (2000) · Educational and Psychological Measurement · doi:10.1177/0013164400605001 · Cited by 66
“Despite ongoing endorsements of its crucial value (Carver, 1978; Cohen, 1990, 1994; Levin, 1998; Lykken, 1968; Rosnow & Rosenthal, 1989; Thompson, 1994b, 1996, 1997), less than 1% of empirical articles published in a sample of psychology journals were replications (Bozarth & Roberts, 1972; Sterling, 1959).”
BackgroundMethodMeta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 61
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 73
The latency of a visual evoked potential tracks the onset of decision making
Nunez, M. D., Gosai, A. K., Vandekerckhove, J., & Srinivasan, R. (2018) · bioRxiv · doi:10.1101/275727 · Cited by 55
Consideraciones prácticas acerca de la detección del sesgo de publicación
Pérez, S. P., & Delgado‐Rodríguez, M. (2006) · Gaceta Sanitaria · doi:10.1157/13101085 · Cited by 74
Öneş, D. S., Viswesvaran, C., & Schmidt, F. L. (2016) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.09.011 · Cited by 56
“Severe problems of questionable research practices have been detailed elsewhere (e.g., Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995; cf. Schmidt & Hunter, 2014, pp. 523–525, Chapter 13).”
BackgroundHéroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 62
“Publication bias—where significant results are more likely to get published—has a long history in science [37].”
BackgroundReliability, fairness, objectivity and other inappropriate goals in peer review
Bailar, J. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065705 · Cited by 56
Do We Really Need More Replications?
Hunt, K. P. (1975) · Psychological Reports · doi:10.2466/pr0.1975.36.2.587 · Cited by 53
“This was a follow-up on a similar survey done by Sterling (1959) who obtained a comparable figure of 97%.”
ResultMeta-analysis, can it ever fail?
Kotiaho, J. S., & Tomkins, J. L. (2002) · Oikos · doi:10.1034/j.1600-0706.2002.960316.x · Cited by 52
On gender and philosophical intuition: Failure of replication and other negative results
Seyedsayamdost, H. (2015) · Philosophical Psychology · doi:10.1080/09515089.2014.893288 · Cited by 52
“…studies that have examined the proportion of positive results of null hypothesis significance testing (NHST) in the published literature have found that between 94% and 97% of articles report positive results (Bozarth & Roberts, 1972; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundResultInfluentialAccuracy of Effect Size Estimates from Published Psychological Research
Brand, A., Bradley, M. T., Best, L. A., & Stoica, G. (2008) · Perceptual and Motor Skills · doi:10.2466/pms.106.2.645-649 · Cited by 52
“Sterling, T. D. (1959).”
Clinical attrition due to biased preclinical assessments of potential efficacy
Lindner, M. D. (2007) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2007.05.002 · Cited by 49
False positive outcomes and design characteristics in occupational cancer epidemiology studies.
Swaen, G., Teggeler, O., & van Amelsvoort, L. V. (2001) · International Journal of Epidemiology · doi:10.1093/ije/30.5.948 · Cited by 52
Button, K. S., Bal, L., Clark, A. H., & Shipley, T. (2016) · BMC Psychology · doi:10.1186/s40359-016-0167-7 · Cited by 62
Metadata
- Authors
- Sterling, T. D.
- Affiliations
- University of Cincinnati (US)
- Corresponding author
- Sterling, Theodore D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1959-03
- Volume
- 54
- Issue
- 285
- Pages
- 30-34
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2062908954
- Cited by
- 964
- References
- 0
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
