The DOI matches its Crossref record.
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract
There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most conclusions in psychology are based on the results of null hypothesis significance testing (NHST; Cumming et al., 2007; Hubbard & Ryan, 2000; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Sterling, 1959
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [12]
In the citing paper
“These causes of overestimation can dramatically increase Type I error rates in the meta-analysis (Sterling, 1959).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Sterling, 1959
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- 1958, Cancer Research 18
- 1957, Publication Manual,
- Edwards A. L., 1950, Experimental Design in Psychological Research
- Lewis D., 1949, Psychological Bulletin, 46, 433, 10.1037/h0059088 · DOI
- Lubin A., 1957, American Psychologist, 8, 519, 10.1037/h0039746 · DOI
- McNemar, 1955, Psychological Statistics,
- Savage L. J., 1954, The Foundations of Statistics
- Walker H. M., 1947, Elementary Statistical Methods
Cited by
Sorted by citations
Items 101–200 of 888
The Insignificance of Null Hypothesis Significance Testing
Gill, J. (1999) · Political Research Quarterly · doi:10.1177/106591299905200309 · Cited by 330
Osborne, J., Simon, S., Christodoulou, A., Howell‐Richardson, C., & Richardson, K. (2013) · Journal of Research in Science Teaching · doi:10.1002/tea.21073 · Cited by 268
“Findings such as these then act to prevent the bias that arises in our understanding of a phenomenon (Lehrer, 2010; Moller & Jennions, 2001) such as that identified by Thomas Sterling who noted in 1959 that 97% of all published studies in psychology found the effect they were looking for with statistical significance…”
BackgroundStatistics in medical journals: developments in the 1980s.
Altman, D. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101206 · Cited by 267
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
“Publication bias arises when the probability that a scientific study is published is not independent of its results [16].”
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
“Authors may assume that they must obtain statistical significance in their studies in order to publish (Sterling, 1959).”
BackgroundMeta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
“An early analysis of 362 articles in psychology found no attempted single replication study for any of them (Sterling, 1959), a subsequent analysis found replications for fewer than 1% of more than 1,000 articles (Bozarth & Roberts, 1972), and a 2012 analysis found replications for 1% of original articles (Makel, Pluc…”
BackgroundInfluentialPublication bias: what is it? How do we measure it? How do we avoid it?
Song, F., Hooper, L., & Loke, Y. K. (2013) · Open Access Journal of Clinical Trials · doi:10.2147/oajct.s34419 · Cited by 298
“…results might be underreported because the results of 97% of psychological studies published in four journals were statistically significant, 35 and the same author in 1995 concluded that the practices leading to publication bias in psychological research had not changed over a period of 30…”
Gerber, A. S., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 260
A Large Scale Test of the Effect of Social Class on Prosocial Behavior
Korndörfer, M., Egloff, B., & Schmukle, S. C. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133193 · Cited by 255
“That is, when combining multiple studies in one article, the persuasiveness and robustness seem to increase, but actually, the power diminishes as more statistical tests are executed [48–53,56–58].”
Peer review for journals: Evidence on quality control, fairness, and innovation
Armstrong, J. S. (1997) · Science and Engineering Ethics · doi:10.1007/s11948-997-0017-3 · Cited by 263
Humphreys, M., Sierra, R., & van der Windt, P. C. (2013) · Political Analysis · doi:10.1093/pan/mps021 · Cited by 303
“There are other benefits to registration, including the ability to assess the population of studies that are never completed or never published. Here, however, we focus on the benefits of ex ante specification of analysis plans. (5)See, for example, Lancet (2010). A mechanism to report observational studies already ex…”
BackgroundLying takes time: A meta-analysis on reaction time measures of deception.
Suchotzki, K., Verschuère, B., Bockstaele, B. V., Ben‐Shakhar, G., & Crombez, G. (2017) · Psychological Bulletin · doi:10.1037/bul0000087 · Cited by 309
“the tendency that large and significant effects are more likely to be published than small and non-significant effects, poses a problem in many research areas and may lead to overestimations of effect sizes in meta-analyses (e.g., Fanelli, 2012; Rosenthal, 1979; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241
Smits, J. A. J., Berry, A. C., Tart, C. D., & Powers, M. B. (2008) · Behaviour Research and Therapy · doi:10.1016/j.brat.2008.06.010 · Cited by 241
Limitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
Tsai, A. C., & Burns, B. F. O. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.06.024 · Cited by 227
“…in our review in that all demonstrated statistically significant associations between one or more psychosocial problems and the outcomes of interest, our summary conclusions could have been driven by non-publication of non-statistically significant findings (Rosenthal, 1979; Sterling, 1959).”
ResultChambers, C., Feredoes, E., Muthukumaraswamy, S., & Etchells, P. (2014) · doi:10.3934/neuroscience2014.1.4 · Cited by 227
Bonovas, S., Filioussi, K., Tsavaris, N., & Sitaras, N. M. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.02.7045 · Cited by 234
Effect of public subsidies on farm technical efficiency: a meta-analysis of empirical results
Minviel, J., & Latruffe, L. (2016) · Applied Economics · doi:10.1080/00036846.2016.1194963 · Cited by 256
“In addition, it has been documented that certain studies remain unpublished because of theoretical or ideological divergences, or conflicts of interest between researchers (Sterling 1959; Mahoney 1977).”
BackgroundHedges, L., & Vevea, J. L. (1996) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986021004299 · Cited by 223
A Handbook of Statistical Analyses using R
Everitt, B., & Hothorn, T. (2006) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420010657 · Cited by 217
A Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213
Peer Review for Journals as it Stands Today—Part 1
Campanario, J. M. (1998) · Science Communication · doi:10.1177/1075547098019003002 · Cited by 200
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 221
“Sterling (1959) noted that 97.3% of statistical tests rejected the null hypothesis for the major scientific findings reported in four psychology journals.”
BackgroundInfluentialAn Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202
A meta-analysis of sex differences in human navigation skills
Nazareth, A., Huang, X., Voyer, D., & Newcombe, N. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01633-6 · Cited by 201
“Such a situation raises the possible influence of the “file-drawer problem” (Sterling, 1959), suggesting that studies producing nonsignificant”
BackgroundGaither, C., & Cavazos-Gaither, A. E. (2018) · The War of Words · doi:10.2307/j.ctvc77m52.73 · Cited by 193
Rape Myth Consistency and Gender Differences in Perceiving Rape Victims
Hockett, J., Smith, S., Klausing, C. D., & Saucier, D. (2016) · Violence against Women · doi:10.1177/1077801215607359 · Cited by 194
“That is, over 671 studies showing null results would be needed to reduce the overall effect size to near-zero levels (Sterling, 1959), suggesting that the overall effect is quite robust to the lack of inclusion of unpublished null effects.”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 345
“For example, selectively reporting experiments or analyses, particularly reporting only those that 'worked', biases the literature by ignoring negative or null results (Fanelli, 2010; Fanelli, 2011; Ioannidis, 2005; Rosenthal, 1979; Sterling, 1959; Sterling et al., 1995).”
BackgroundAn historical perspective on meta-analysis: Dealing quantitatively with varying study results
O’Rourke, K. (2007) · Journal of the Royal Society of Medicine · doi:10.1177/0141076807100012020 · Cited by 216
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 208
A Meta–Analytic Review of Psychological Treatments for Social Anxiety Disorder
Powers, M. B., Sigmarsson, S. R., & Emmelkamp, P. M. G. (2008) · International Journal of Cognitive Therapy · doi:10.1521/ijct.2008.1.2.94 · Cited by 191
Diabetes mellitus as a risk factor for primary open‐angle glaucoma: a meta‐analysis
Bonovas, S., Peponis, V., & Filioussi, K. (2004) · Diabetic Medicine · doi:10.1111/j.1464-5491.2004.01173.x · Cited by 186
Beyond MEDLINE for Literature Searches
Conn, V. S., Isaramalai, S., Rath, S., Jantarakupt, P., Wadhawan, R., & Dash, Y. (2003) · Journal of Nursing Scholarship · doi:10.1111/j.1547-5069.2003.00177.x · Cited by 184
Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175
Review Paper : Psychology of the Scientist: An Evaluative Review
Mahoney, M. J. (1979) · Social Studies of Science · doi:10.1177/030631277900900304 · Cited by 180
PUBLICATION BIAS: THE ACHILLES’ HEEL OF SYSTEMATIC REVIEWS?
Torgerson, C. (2006) · British Journal of Educational Studies · doi:10.1111/j.1467-8527.2006.00332.x · Cited by 178
“In his early cross-sectional study Sterling (1959) demonstrated that, in four major psychological journals published in 1955 and 1956, there was a greater probability of the results of experiments being published if the relevant test of significance rejected the null hypothesis than if the test…”
BackgroundResearch on scientific journals: Implications for editors and authors
Armstrong, J. S. (1982) · Journal of Forecasting · doi:10.1002/for.3980010109 · Cited by 174
“Sterling (1959) also found evidence of the preference for a dominant hypothesis: His review of research in four commonly read psychology journals found that 97 percent of the papers confirmed the dominant hypothesis (at p 0.05).”
InfluentialPublication Bias in Empirical Sociological Research
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 175
Threats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. A. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 176
Improving Doctors' Understanding of Statistics
Altman, D. G., & Bland, J. M. (1991) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983040 · Cited by 193
Could It Be Better to Discard 90% of the Data? A Statistical Paradox
Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010) · The American Statistician · doi:10.1198/tast.2009.08205 · Cited by 170
“…publication selection, ‘publication selection bias,’ has long been acknowledged as a severe threat to statistical inference and scientific practice (Sterling 1959; Rosenthal 1979; Hedges and Olkin 1985; Begg and Berlin 1988; Sterling, Rosenbaum, and Weinkam 1995; and Copas 1999, to mention a…”
BackgroundThe New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 183
“The file drawer problem has been well-documented but unsolved for almost 60 years (Sterling 1959; Sterling et al. 1995).”
BackgroundA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Replication, statistical consistency, and publication bias
Francis, G. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.02.003 · Cited by 126
“For example, as noted by Sterling (1959), Fanelli (2010), and others, the field of psychology has a preference for publishing statistically significant findings.”
BackgroundCockburn, A., Gutwin, C. A., & Dix, A. J. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173715 · Cited by 109
“3173715 of NHST have been raised for nearly sixty years [37].”
BackgroundThe importance of negative results in psychological research.
Smart, R. (1964) · The Canadian Psychologist · doi:10.1037/h0083036 · Cited by 156
Reliability of reviews for the American Psychologist: A biostatistical assessment of the data.
Cicchetti, D. V. (1980) · American Psychologist · doi:10.1037/0003-066x.35.3.300 · Cited by 161
Problems induced by meta‐analyses
Chalmers, T. C. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100618 · Cited by 144
Power Posing: P- Curving the Evidence
Simmons, J. P., & Simonsohn, U. (2017) · Psychological Science · doi:10.1177/0956797616658563 · Cited by 179
“Statistically significant results are more likely to be published than results that are not significant (Greenwald, 1975; Rosenthal, 1979; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundBorle, P., Reichel, K., Niebuhr, F., & Voelter‐Mahlknecht, S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168673 · Cited by 199
“, the non-publication of non-statistically significant findings [49,50].”
BackgroundEstimation for Better Inference in Neuroscience
Calin-Jageman, R. J., & Cumming, G. (2019) · eNeuro · doi:10.1523/eneuro.0205-19.2019 · Cited by 147
“This implausible “excess of significance” indicates that many unfavorable results have been discarded to the file drawer (Sterling, 1959; Sterling et al., 1995) or inappropriately coaxed under the threshold for statistical significance (Simmons et al.”
BackgroundDalton, D. R., Aguinis, H., Dalton, C. M., Bosco, F. A., & Pierce, C. A. (2012) · Personnel Psychology · doi:10.1111/j.1744-6570.2012.01243.x · Cited by 157
Schmidt, F. L., & Oh, I.-S. (2016) · Archives of Scientific Psychology · doi:10.1037/arc0000029 · Cited by 145
A Proposal for a New Editorial Policy in the Social Sciences
Walster, G. W., & Cleary, T. A. (1970) · The American Statistician · doi:10.1080/00031305.1970.10478884 · Cited by 144
Meta-Regression Analysis as the Socio-Economics of Economic ResearchPreprint
Doucouliagos, C., Stanley, T., & Jarrell, S. B. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1014307 · Cited by 141
Ambient Particulate Matter and Health Effects
Anderson, H. R., Atkinson, R. W., Peacock, J. L., Sweeting, M. J., & Marston, L. (2005) · Epidemiology · doi:10.1097/01.ede.0000152528.22746.0f · Cited by 124
Howard, G., Maxwell, S., & Fleming, K. J. (2000) · Psychological methods · doi:10.1037/1082-989x.5.3.315 · Cited by 141
Perioperative blood transfusion and cancer recurrence: meta‐analysis for explanation
Vamvakas, E. C. (1995) · Transfusion · doi:10.1046/j.1537-2995.1995.35996029162.x · Cited by 141
History repeating: guidelines to address common problems in psychedelic science
van Elk, M., & Fried, E. I. (2023) · Therapeutic Advances in Psychopharmacology · doi:10.1177/20451253231198466 · Cited by 137
“More generally, we hope to see more nuanced conclusions in scientific studies, university press releases, and media reports; the current hype directly contributes to validity threats for psychedelic science and does a strong disservice to treatment-seeking populations.”
BackgroundWorsening file-drawer problem in the abstracts of natural, medical and social science databases
Pautasso, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0233-5 · Cited by 136
“The file-drawer problem is the relative lack of non-significant published results (e.g., Sterling 1959; Begg and Berlin 1988; Kennedy 2004).”
ResultLessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. J. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 174
Is once enough?: on the extent and content of replications in human-computer interaction
Hornbæk, K., Sander, S., Bargas-Avila, J. A., & Simonsen, J. G. (2014) · International Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557004 · Cited by 133
“For example, Sterling [46] found no replications in a sample of 362 articles from psychology journals.”
BackgroundA kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
“Substituting g(·) in (2) by its EK approximation (5) gives the EK meta-regression model:”
BackgroundInfluentialDetecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132
“The first form of this effect that was acknowledged was publication bias (Sterling, 1959; Simes, 1986; Begg and Berlin, 1988; Berlin et al., 1989; Rust et al., 1990; Sterling et al., 1995; Dickersin, 1990; Easterbrook et al., 1991; Sutton et al., 2000a).”
BackgroundData-Dredging Procedures in Survey Analysis
Selvin, H., & Stuart, A. (1966) · The American Statistician · doi:10.1080/00031305.1966.10480401 · Cited by 132
Finding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131
Mychailyszyn, M. P., Brodman, D. M., Read, K. L., & Kendall, P. C. (2012) · Clinical Psychology Science and Practice · doi:10.1111/j.1468-2850.2012.01279.x · Cited by 141
“Validity Assessment The standards of peer review have led to the suspicion that published studies are more likely to have statistically significant findings than those that are not published (Sterling, 1959).”
BackgroundUsing Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials.
McLeod, B. D., & Weisz, J. R. (2004) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.72.2.235 · Cited by 125
“, Atkinson, Furlong, & Wampold, 1982; Coursol & Wagner, 1986) and that authors tend to file away research that does not generate statistically significant results (i.e., file drawer effect—see, e.g., Rosenthal, 1979; Sterling, 1959).”
BackgroundMethodLinden, A. H., & Hönekopp, J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964193 · Cited by 159
“This precludes distortions of their effect sizes through publication bias and questionable research practices, which might often affect other published results (Ferguson & Brannick, 2012; Kühberger et al., 2014; LeBel & Peters, 2011; Levine et al., 2009; Pashler & Harris, 2012; Sterling, 1959).”
BackgroundResultInfluentialMotor expertise and performance in spatial tasks: A meta-analysis
Voyer, D., & Jansen, P. (2017) · Human Movement Science · doi:10.1016/j.humov.2017.04.004 · Cited by 140
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
Kepes, S., & McDaniel, M. A. (2013) · Industrial and Organizational Psychology · doi:10.1111/iops.12045 · Cited by 136
Does the UK Minimum Wage Reduce Employment? A Meta‐Regression Analysis
Leonard, M. D. L., Stanley, T. D., & Doucouliagos, H. (2013) · British Journal of Industrial Relations · doi:10.1111/bjir.12031 · Cited by 94
“Publication selection is a widely accepted fact in the social science, medical research, and economics (Sterling, 1959; Tullock, 1959; Feige, 1975; Rosenthal, 1979; Glass, McGaw and Smith, 1981; Lovell, 1983; Hedges and Oklin, 1985; Begg and Berlin, 1988; DeLong and Lang, 1992; Card and Krueger, 1995b).”
Atkinson, D. R., Furlong, M., & Wampold, B. (1982) · Journal of Counseling Psychology · doi:10.1037/0022-0167.29.2.189 · Cited by 120
The Impact of Age and Education on Response Order Effects in Attitude Measurement
Knäuper, B. (1999) · Public Opinion Quarterly · doi:10.1086/297724 · Cited by 118
Meta-analysis: Techniques, applications, and functions.
Strube, M., & Hartmann, D. P. (1983) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.51.1.14 · Cited by 113
Pair-bonding, fatherhood, and the role of testosterone: a meta-analytic review
Grebe, N. M., Sarafin, R. E., Strenth, C. R., & Zilioli, S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.010 · Cited by 116
“Publication bias, the increased chance for statistically significant findings to be published compared to non-significant effects, is one of the oldest and most well-known sources (Sterling, 1959).”
DIFFERENTIAL RECALL BIAS AND SPURIOUS ASSOCIATIONS IN CASE/CONTROL STUDIES
Barry, D. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19961215)15:23 · Cited by 39
The Initial Examination of Data
Chatfield, C. (1985) · doi:10.2307/2981969 · Cited by 116
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Nosek, B. A., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 136
“Moreover, studies with null findings are scarce in the published research base, especially in the social sciences (Fanelli, 2010; Sterling, 1959).”
BackgroundMetastudies for robust tests of theory
Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C., Boeck, P. D., & Vandekerckhove, J. (2018) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1708285114 · Cited by 115
“This statistical significance filter (5) biases the published record toward results that capitalize on measurement noise and fluke outcomes (6).”
BackgroundConcepts of Relative Importance in Recent Scientific Literature
Kruskal, W., & Majors, R. (1989) · The American Statistician · doi:10.1080/00031305.1989.10475596 · Cited by 115
Davis, M. L., Powers, M. B., Handelsman, P., Medina, J. L., Zvolensky, M., & Smits, J. A. J. (2015) · Evaluation & the Health Professions · doi:10.1177/0163278714529970 · Cited by 114
“Several authors suggest there may be a potential discrepancy between the number of published trials and the total number that are completed (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959).”
BackgroundReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Flore, P. (2019) · The Federal Estate Tax · doi:10.7551/mitpress/10365.003.0010 · Cited by 113
“Publication bias refers to the practice of primarily publishing articles in which significant results are shown, thus leaving the so-called null results in the file drawer (Ioannidis, 2005; Rosenthal, 1979; Sterling, 1959), a practice that can lead to serious inflations of estimated effect-sizes in meta-analyses (Bakk…”
Bonovas, S., Filioussi, K., Tsantes, A., & Peponis, V. (2004) · Public Health · doi:10.1016/j.puhe.2003.09.009 · Cited by 110
Unbiased, reliable, and valid student evaluations can still be unfair
Esarey, J., & Valdes, N. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2020.1724875 · Cited by 159
“…Cohen (1981) and Feldman (1989) and find that their results are influenced by the tendency for small studies with exaggeratedly positive results to be published while null results are ignored (Schwarzer et al., 2015, Chapter 5; see also Sterling, 1959; Sterling et al., 1995; Franco et al., 2014).”
BackgroundChow's defense of null-hypothesis testing: Too traditional?
Glück, J., & Vitouch, O. (1998) · Behavioral and Brain Sciences · doi:10.1017/s0140525x9840116x · Cited by 109
“Significance testing has become widespread in psychology (e.g., Hubbard et al. 1997; Sterling 1959).”
The Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing
Lash, T. L. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx261 · Cited by 128
Dickersin, K., & Chalmers, I. (2011) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2011.11k042 · Cited by 149
Chinazzo, G., Andersen, R. K., Azar, E., Barthelmes, V. M., Becchio, C., Belussi, L., Berger, C., Carlucci, S., Corgnati, S. P., Crosby, S., et al. (2022) · Building and Environment · doi:10.1016/j.buildenv.2022.109719 · Cited by 127
Statistical reform in medicine, psychology and ecology
Fidler, F. M., Cumming, G., Burgman, M. A., & Thomason, N. R. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.035 · Cited by 95
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G., & Clarke, R. M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 116
“Sterling (1959) found in his review of 362 psychology articles that none were replications; Bozarth and Roberts (1972) similarly found that of 1046 clinical/counseling psychology papers published between 1967 and 1970, 1% were replications.”
BackgroundFurchtgott, E. (1984) · American Psychologist · doi:10.1037/0003-066x.39.11.1315.b · Cited by 106
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
Yom, S. L. (2014) · Comparative Political Studies · doi:10.1177/0010414014554685 · Cited by 148
“Another pathology lays in the selective reporting of positive findings while ignoring conflicting or null results, especially when running multiple tests (Rosenthal, 1979; Sterling, 1959).”
BackgroundThe causes, consequences and detection of publication bias in psychiatry
Gilbody, S., Song, F.-J., Eastwood, A., & Sutton, A. (2000) · Acta Psychiatrica Scandinavica · doi:10.1034/j.1600-0447.2000.102004241.x · Cited by 105
“Despite knowing about the existence of publication bias for some 40 years (41), it remains as prevalent now as it was in 1959 (42).”
Background
Metadata
- Authors
- Sterling, T. D.
- Affiliations
- University of Cincinnati (US)
- Corresponding author
- Sterling, Theodore D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 1959-03
- Volume
- 54
- Issue
- 285
- Pages
- 30-34
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2062908954
- Cited by
- 964
- References
- 0
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
- FWCI
- 9.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Statistical Association (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
