Cited by 964Meta-analysis and systematic reviewsAcademic integrity and plagiarismAcademic Publishing and Open Access

The DOI matches its Crossref record.

Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa

Sterling, Theodore D.·1959·Journal of the American Statistical Association

DOI 10.1080/01621459.1959.10501497

Open this paper in CoChat — free account

Abstract

There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.

Integrity signals

Registry

Crossref · 10.1080/01621459.1959.10501497

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

No references could be resolved yet.

Research works printed but not linked: 8

1958, Cancer Research 18
1957, Publication Manual,
Edwards A. L., 1950, Experimental Design in Psychological Research
Lewis D., 1949, Psychological Bulletin, 46, 433, 10.1037/h0059088 · DOI
Lubin A., 1957, American Psychologist, 8, 519, 10.1037/h0039746 · DOI
McNemar, 1955, Psychological Statistics,
Savage L. J., 1954, The Foundations of Statistics
Walker H. M., 1947, Elementary Statistical Methods

Sorted by citations

Items 501–600 of 888

Filled Pauses Produced by Autistic Adults Differ in Prosodic Realisation, but not Rate or Lexical Type
Wehrle, S., Grice, M., & Vogeley, K. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-023-06000-y · Cited by 14
“…to date, especially when taking into account the serious and pertinent issue of publication bias (whereby studies that find a “significant” effect are vastly more likely to be published than those that do not; DeVito & Goldacre, 2019; Easterbrook et al., 1991; John et al., 2012; Sterling, 1959).”
Background
Sorting the File Drawer: A Typology for Describing Unpublished Studies
Lishner, D. A. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620979831 · Cited by 24
“The concept of the file drawer was developed to assist in framing discussions of earlier concerns that published studies may be unrepresentative of the total population of studies conducted by scientists (McNemar, 1960; Smart, 1964; Sterling, 1959).”
Bayesian mixture modeling of significant p values: A meta-analytic method to estimate the degree of contamination from H₀.
Gronau, Q. F., Duizer, M., Bakker, M., & Wagenmakers, E. (2017) · Journal of Experimental Psychology General · doi:10.1037/xge0000324 · Cited by 13
“Hence, if the observed distribution of significant p values is relatively flat, the most likely explanation for the findings is publication bias (e.g., Rosenthal, 1979; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Background
Significance Testing Needs a Taxonomy
Bradley, M. T., & Brand, A. (2016) · Psychological Reports · doi:10.1177/0033294116662659 · Cited by 14
L'usage des tests statistiques par les chercheurs en psychologie : aspects normatif, descriptif et prescriptif
Poitevineau, J. (2004) · Mathématiques et sciences humaines · doi:10.4000/msh.2900 · Cited by 14
Peer Review: From “Sacred Ideals” to “Profane Realities”
Johnson, D. R., & Hermanowicz, J. C. (2017) · Higher education · doi:10.1007/978-3-319-48983-4_10 · Cited by 46
Preregistration in diverse contexts: a preregistration template for the application of cognitive models
Crüwell, S., & Evans, N. J. (2021) · Royal Society Open Science · doi:10.1098/rsos.210155 · Cited by 28
Trialling Meta-Research in Comparative Cognition: Claims and Statistical Inference in Animal Physical Cognition
Farrar, B. G., Altschul, D., Fischer, J., van der Mescht, J., Placì, S., Troisi, C. A., Vernouillet, A., Clayton, N. S., & Ostojić, L. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.03.09.2020 · Cited by 17
“Publication bias leads to a literature filled with overestimated effect sizes (Hedges, 1984; Sterling, 1959), and facilitates the canonization of false facts (Nissen et al., 2016).”
Background
Changing the logic of replication: A case from infant studies.Preprint
Margoni, F., & Shepperd, M. (2019) · Infant Behavior and Development · doi:10.31234/osf.io/xw6qt · Cited by 13
Comparative assessment of the intrinsic sensitivity of crop species and wild plant species to plant protection products and their active substances and potential implications for the risk assessment: A literature review
Christl, H., Morilla, J., Hoen, T., & Zumkier, U. (2019) · Integrated Environmental Assessment and Management · doi:10.1002/ieam.4115 · Cited by 13
“…bias occurs when the outcomeof an experiment or research study influences thedecisionwhether topublish (or otherwise distribute) it (e.g., Sterling 1959; Dickersin 1990, 1997; Ioannidis 2005; Song et al. 2013; MP Zeegers, CAPHRI School for Public Health and Primary Care, Maastricht…”
BackgroundResult
Excess Success in “Ray of hope: Hopelessness Increases Preferences for Brighter Lighting”
Francis, G., & Thunell, E. (2019) · Collabra: Psychology · doi:10.1525/collabra.213 · Cited by 13
Estimating statistical power, posterior probability and publication bias of psychological research using the observed replication rate
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 22
“A series of observations spanning five decades has indicated that >90% published studies in psychology reported positive findings, that is, where the authors’ hypothesis was supported by data (Fanelli, 2010; T. D. Sterling, Rosenbaum, & Weinkam, 1995; Theodore D. Sterling, 1959).”
Result
Research synthesis methods and library and information science: Shared problems, limited diffusion
Sheble, L. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23499 · Cited by 16
“Preferential evaluations of some research, most commonly that which is statistically significant (e.g., Fanelli, 2012, 2013; Pautasso, 2010; Schneider, 2013, Sterling, 1959), contributes to this problem.”
Background
On the Size of Sheepskin Effects: A Meta-Analysis
Rodríguez, J. J. M., & Muro, J. (2015) · Economics · doi:10.5018/economics-ejournal.ja.2015-37 · Cited by 13
“Meta-analysis has been used in medical and psychological studies on a regular basis (see, e.g., Sterling, 1959; Rosenthal, 1979; Begg and Berlin.”
Method
Introducing Prospective Manuscript Review to Address Publication Bias
Mell, L. K., & Zietman, A. L. (2014) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2014.07.052 · Cited by 13
Unpublishable research: examining and organizing the ‘file drawer’
Tsou, A., Schickore, J., & Sugimoto, C. R. (2014) · Learned Publishing · doi:10.1087/20140404 · Cited by 13
Das Signifikanztest-Ritual und andere Sackgassen des Fortschritts in der Statistik
Krämer, W. (2012) · AStA Wirtschafts und Sozialstatistisches Arch. · doi:10.1007/s11943-012-0110-1 · Cited by 13
Some suggestions from sociology of science to advance the psi debate
Pinch, T. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054832 · Cited by 13
Considerations for fitting occupancy models to data from eBird and similar volunteer-collected data
Hochachka, W., Ruiz-Gutierrez, V., & Johnston, A. (2023) · The AUK: A Quarterly Journal of Ornithology · doi:10.1093/ornithology/ukad035 · Cited by 12
Incentive-Theoretic Bayesian Inference for Collaborative Science
Bates, S., Jordan, M. I., Sklar, M., & Soloff, J. (2023) · arXiv.org · doi:10.48550/arxiv.2307.03748 · Cited by 12
“Classically, Sterling (1959); Tullock (1959) point out that when only positive findings are published, the total false positive rate of reported results in the scientific literature may be large.”
Background
Critical Values Robust to P-Hacking
McCloskey, A., & Michaillat, P. (2024) · The Review of Economics and Statistics · doi:10.1162/rest_a_01456 · Cited by 11
Analytic Transparency, Radical Honesty, and Strategic Incentives
Yom, S. (2018) · PS · doi:10.1017/s1049096517002554 · Cited by 12
“Sterling, Theodore.”
Background
Atomoxetine treatment for children and adolescents with Attention-Deficit/Hyperactivity Disorder (ADHD): a comprehensive meta-analysis of outcomes on parent-rated core symptomatology
Gayleard, J., & Mychailyszyn, M. P. (2017) · ADHD Attention Deficit and Hyperactivity Disorders · doi:10.1007/s12402-017-0216-y · Cited by 12
A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes
Crosetto, P., Filippin, A., & Heider, J. (2014) · CESifo Economic Studies · doi:10.1093/cesifo/ifu029 · Cited by 9
“The fact that published results may not be a representative sample of all scientific studies is something that has long been debated in the literature starting from Sterling (1959).”
Background
Publication Bias in Recent Empirical Accounting ResearchPreprint
Basu, S., & Park, H. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2379889 · Cited by 23
Principles of Applied Statistics
Cox, D., & Donnelly, C. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9781139005036 · Cited by 12
“The effect, recognized for many years (Sterling, 1959; Dawid and Dickey, 1977), was labelled the “file drawer effect” in 1979 (Rosenthal, 1979) and, although files are now more likely to be computer- rather than drawerbased, the phenomenon remains.”
Do we really want more “reliable” reviewers?
KRAEMER, H. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065870 · Cited by 11
Referee agreement in context
Hargens, L. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065857 · Cited by 12
Psi, statistics, and society
Utts, J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054923 · Cited by 12
The Effect of Professional Advice on the Stability of a Speculative Market
Denton, F. (1985) · Journal of Political Economy · doi:10.1086/261345 · Cited by 12
Relations between some Components of Attraction-to-Group
Ramuz-Nienhuis, W., & van Bergen, A. V. (1960) · doi:10.1177/001872676001300307 · Cited by 12
Optimal Estimation When Researcher and Social Preferences Are Misaligned
Spiess, J. (2025) · Econometrica · doi:10.3982/ecta18640 · Cited by 12
“The resulting estimator resembles Wager et al.’s (2016) “cross-estimation” and Chernozhukov et al.”
Are Difficult-To-Study Populations too Difficult to Study in a Reliable Way?
Lange, F. (2019) · European Psychologist · doi:10.1027/1016-9040/a000384 · Cited by 19
Conventional wisdom, meta‐analysis, and research revision in economics
Gechert, S., Mey, B., Opatrný, M., Havránek, T., Stanley, T. D., Bom, P. R. D., Doucouliagos, H., Heimberger, P., Iršová, Z., & Rachinger, H. J. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12630 · Cited by 11
Crossing the Red Line? Empirical Evidence and Useful Recommendations on Questionable Research Practices among Business Scholars
Latan, H., Jabbour, C. J. C., Jabbour, A. B. L. D. S., & Ali, M. (2021) · Journal of Business Ethics · doi:10.1007/s10551-021-04961-7 · Cited by 14
“In his initial study on this topic, Sterling (1959) illustrates how the practice of selective reporting of findings (i.e., only reporting statistically significant results) leads to incorrect conclusions.”
BackgroundInfluential
Can ecotourism interpretation influence reef protective behaviours? Findings from a quasi-experimental field study involving a virtual reality game
Coghlan, A. (2021) · Journal of Ecotourism · doi:10.1080/14724049.2021.1971240 · Cited by 11
“…to changing PEBs along with the decision to measure actual behaviour, rather than behavioural intent or self-reported behaviour are highlighted, alongside the issue of publication bias, where the probability that a scientific study is published is not independent of its results (Sterling, 1959).”
Background
Comprehensive Appraisal
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1
Editorial: P-values, theory, replicability, and rigour
Bijak, J. (2019) · Demographic Research · doi:10.4054/demres.2019.41.32 · Cited by 11
“…a misuse or misinterpretation of statistical methods can lead to “false positives, overhyped claims and overlooked effects” (Nature Editorial 2019: 283), creating publication bias against ‘non-significant’ or negative results (for an early warning, see Sterling 1959 and, recently, Fanelli 2012).”
Background
p-Hacking in Experimental Audit Research
Khan, M. J., & Tronnes, P. C. (2018) · Behavioral Research in Accounting · doi:10.2308/bria-52183 · Cited by 16
“…in the p-value distribution is also consistent with the notion that researchers in the experimental audit discipline, similar to other experimental sciences, may be responding to the pressure of reporting statistical significance in order to improve the likelihood of publication (Sterling, 1959).”
BackgroundResult
When Does Partitioned Pricing Lead to More Favorable Consumer Preferences?: Meta-Analytic Evidence:
Abraham, A. T., & Hamilton, R. W. (2018) · Journal of Marketing Research · doi:10.1509/jmr.14.0567 · Cited by 11
A Monte-Carlo Estimation of Effect Size Distortion Due to Significance Testing
Bradley, M. T., Smith, D., & Stoica, G. (2002) · Perceptual and Motor Skills · doi:10.2466/pms.2002.95.3.837 · Cited by 11
Sequential Anovas and Type I Error Rates
Mccarroll, D., Crays, N., & Dunlap, W. P. (1992) · Educational and Psychological Measurement · doi:10.1177/0013164492052002014 · Cited by 10
What should be done improve reviewing?
Crandall, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065766 · Cited by 10
Disagreement among journal reviewers: No cause for undue alarm
Stricker, L. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065985 · Cited by 11
How Significant is Statistical Significance? A critique of the use of statistics in research
French, S. (1988) · doi:10.1016/s0031-9406(10)63189-6 · Cited by 11
“Sterling (1970) found that failure to publish statistically non-significant results or replicated studies is very common in a wide range of journals.”
Background
Neuroscience and psi-ence
Beyerstein, B. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054546 · Cited by 11
What Fuels Publication Bias?
Auspurg, K., & Hinz, T. (2011) · Jahrbücher für Nationalökonomie und Statistik · doi:10.1515/jbnst-2011-5-607 · Cited by 17
Parapsychology is science, but its findings are inconclusive
Akers, C. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054480 · Cited by 11
Publication bias in psychology: A closer look at the correlation between sample size and effect size
Linden, A. H., Pollet, T. V., & Hönekopp, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297075 · Cited by 20
“The erroneous belief that statistically non-significant findings are uninformative incentivises researchers to publish statistically significant findings [1,2].”
Background
Evaluating Implementation of the Transparency and Openness Promotion Guidelines: Reliability of Instruments to Assess Journal Policies, Procedures, and Practices
Kianersi, S., Grant, S., Naaman, K., Henschel, B., Mellor, D. T., Apte, S., Deyoe, J. E., Eze, P., Huo, C., Lavender, B. L., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221149735 · Cited by 11
“, 2016), and scientists have been concerned for decades about the lack of replication in the literature (Greenwald, 1976; Nosek et al., 2022; Open Science Collaboration, 2015; Rosenthal, 1979; Sterling, 1959).”
Background
Infant speech perception and cognitive skills as predictors of later vocabulary.
Wang, Y.-Y., Seidl, A., & Cristia, A. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101524 · Cited by 10
An Interactionist Approach to Cognitive Debiasing
Bland, S. (2020) · Episteme · doi:10.1017/epi.2020.9 · Cited by 10
“This longstanding publication bias against inconclusive and replication studies has created a file-drawer problem: the scientific literature likely does not accurately reflect the state of evidence for its positive claims because negative evidence does not get publicized (Sterling 1959).”
Background
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15
Characteristics of mental health trials registered in ClinicalTrials.gov
Arnow, K. D., King, A. C., & Wagner, T. H. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112552 · Cited by 27
Nicht replizieren: publizieren!?
Rost, D. H., & Bienefeld, M. A. (2019) · Zeitschrift für Pädagogische Psychologie · doi:10.1024/1010-0652/a000253 · Cited by 13
Towards a replication culture in phonetic research: Speech production research in the classroomPreprint
Roettger, T. B., & Baer-Henney, D. (2017) · Phonological Data and Analysis · doi:10.17605/osf.io/9kywf · Cited by 10
“Additionally, public preregistration can at least help to reduce issues related to publication bias (Sterling 1959), as the number of failed attempts to reject a hypothesis can be tracked transparently.”
Why prudence is needed when interpreting articles reporting clinical trial results in mental health
Dal‐Ré, R., Bobes, J., & Cuijpers, P. J. (2017) · Trials · doi:10.1186/s13063-017-1899-2 · Cited by 11
“[4] showed that 97% of all published studies on psychology rejected the null hypothesis.”
Background
Ethics in context: a scandal in academia
Heron, M. J., & Belford, P. (2014) · Computer Science On-line Conference · doi:10.1145/2656870.2656875 · Cited by 10
“Sterling [47] found out that 97% of the studies in four major psychology journals had shown positive results at a statistically significant level – only 3% of the studies reported on negative results.”
Background
Four editorial proposals to improve social sciences research and publication
Joseph, J., & Baldwin, S. (2000) · International Journal of Risk & Safety in Medicine · doi:10.3233/jrs-2000-218 · Cited by 9
ESP and the Big Stuff
Glymour, C. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054716 · Cited by 10
Generalizability of choice architecture interventions
Szászi, B., Goldstein, D. G., Soman, D., & Michie, S. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00471-9 · Cited by 15
“However, for several potential reasons, including the intricacies of memory recall 9 , the dynamics of scientific publishing 10 , and the canonical successful examples used in the early ‘nudge’ discourse, it might be more challenging to recollect failed interventions or studies that provide a more…”
An Inception-Cohort Study Quantifying How Many Registered Studies Are Publicly Shared
Ensinck, E. N. F., & Lakens, D. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241296031 · Cited by 9
The Replication Database: Documenting the Replicability of Psychological Science
Röseler, L., Kaiser, L., Doetsch, C., Klett, N., Seida, C., Schütz, A., Aczél, B., Adelina, N., Agostini, V., Alarie, S., et al. (2024) · Journal of Open Psychology Data · doi:10.5334/jopd.101 · Cited by 9
“By using a public and crowdsourced database for replications, researchers may further circumvent journal gatekeeping (Mynatt et al., 1977; Sterling, 1959).”
Method
Estimating the extent of selective reporting: An application to economics
Bruns, S. B., Deressa, T. K., Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1711 · Cited by 9
“21,22 Hacking the p -value to be statistically significant is eased with researchers' degrees of freedom in the analysis being often vast in both observational research 23 – 26 and experimental research.”
The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward
Keener, S. K., Kepes, S., & Torka, A. (2023) · Acta Psychologica · doi:10.1016/j.actpsy.2023.104005 · Cited by 12
Meta-analysis of the effects of sulfate versus hydroxy trace mineral source on nutrient digestibility in dairy and beef cattle.
Ibraheem, M., Kvidera, S., Fry, R. S., & Bradford, B. J. (2023) · Journal of Dairy Science · doi:10.3168/jds.2022-22490 · Cited by 9
Dissecting the tension of open science standards implementation in management and organization journals
Hensel, P. G. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1981870 · Cited by 16
Material Scarcity and Unethical Economic Behavior: A Systematic Review and Meta-AnalysisPreprint
Elbaek, C. T., Mitkidis, P., Aarøe, L., & Otterbring, T. (2021) · Research Square · doi:10.21203/rs.3.rs-800481/v1 · Cited by 9
“) to minimize the possibility of solely obtaining published research, which 16 often tends to favor statistically significant results (Atkinson et al., 1982; Sterling, 1959; 17 Sterling et al., 1995).”
BackgroundInfluential
Standard Errors in Quantitative Criminology: Taking Stock and Looking Forward
Sweeten, G. (2020) · Journal of Quantitative Criminology · doi:10.1007/s10940-020-09463-9 · Cited by 17
“Sterling (1959) reported that 97% of experiments reported in four top psychology journals had rejected their null hypotheses.”
Background
Understanding climate-induced migration through computational modeling: A critical overview with guidance for future efforts
Till, C., Haverkamp, J., White, D., & Bhaduri, B. (2018) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512916679038 · Cited by 9
“The acknowledgement that a publication bias toward significant or targeted results exists has been known since the late 1950s, when Sterling investigated publishing decisions.(98) This was revisited in 1995 by Sterling and colleagues and the original findings still held true.”
Background
The (non-)replicability of regulatory resource depletion: A field report employing non-invasive brain stimulation
Emmerling, F., Martijn, C., Alberts, H., Thomson, A. C., David, B., Kessler, D., Schuhmann, T., & Sack, A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0174331 · Cited by 9
Group Analysis in the Time of Austerity: Neo-liberalism, Managerialism and Evidence-Based Research
Dalal, F. (2017) · Group Analysis · doi:10.1177/0533316416682153 · Cited by 9
International Trade, Do Institutions matter? Evidence from Regional Studies
Ojeaga, P., George, E., Alege, P., & Ogundipe, A. (2014) · doi:10.4236/tel.2014.47074 · Cited by 9
Meta-Analyses
Dalton, D. R., & Dalton, C. M. (2007) · Organizational Research Methods · doi:10.1177/1094428107304409 · Cited by 34
“Although there have been spirited exchanges about the overreliance on these NHST methods for nearly 50 years (Sterling, 1959; also, for excellent overviews, see Hagan, 1997; Harlow, Mulaik, & Steiger, 1997; Howard, Maxwell, & Fleming, 2000; Rosenthal, Rosnow, & Rubin, 2000, also provide a unique…”
Why is the reliability of peer review so low?
Laming, D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065882 · Cited by 7
“Please wait to be tolerated”: Distinguishing fact from fiction on both sides of a scientific controversy
Hövelmann, G. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0005473x · Cited by 9
Where lies the bias
Palmer, J., & Rao, K. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054960 · Cited by 9
Enhancing Rigor in Quantitative Meta-Analyses for Mindfulness Research: A Comprehensive Guide
Sperling, E. L., Khoury, B., Sutton, A., Price‐Blackshear, M. A., & Bettencourt, B. A. (2025) · Mindfulness · doi:10.1007/s12671-025-02517-8 · Cited by 8
“For example, over the course of 60 years of research in psychology, more than 95% of published studies reported positive findings (Scheel et al., 2021; Sterling, 1959).”
A registered report of preregistration practices in studies of electroencephalogram (EEG) and event-related potentials (ERPs): A first look at accessibility, adherence, transparency, and selection bias
Clayson, P. E., Carbine, K. A., Shuford, J. L., McDonald, J. B., & Larson, M. J. (2025) · Cortex · doi:10.1016/j.cortex.2025.02.008 · Cited by 11
A perspective on epigenomic aging processes in the human brain and their plasticity in patients with mental disorders – a systematic review
Postberg, J., Schubert, M. T., Nin, V., Wagner, L., & Piefke, M. (2024) · Neurogenetics · doi:10.1007/s10048-024-00771-x · Cited by 9
“This can lead to a misinterpretation of the evidence [34].”
Background
Position: Embracing Negative Results in Machine Learning
Karl, F., Kemeter, L. M., Dax, G., & Sierak, P. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2406.03980 · Cited by 8
“Publication bias is a known quantity in science and has long been studied (Sterling, 1959; Boulesteix et al., 2015; Ritchie, 2021); perhaps most famously in medical and clinical science (Easterbrook et al., 1991).”
Registered Reports in Child Development : Introduction to the Special Section
Syed, M., Frank, M. C., & Roisman, G. I. (2023) · Child Development · doi:10.1111/cdev.14003 · Cited by 8
Beyond statistical significance: Five principles for the new era of data analysis and reporting
Wedel, M., & Gal, D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1379 · Cited by 12
“Therefore, the (unrealistic) expectation that the p-value for each experiment in a sequence indicates “significance” increases the file drawer problem, whereby studies without “significant” results are not reported (Sterling, 1959).”
How well are open science practices implemented in industrial and organizational psychology and management?
Torka, A., Mazei, J., Bosco, F. A., Cortina, J. M., Götz, M., Kepes, S., O’Boyle, E. H., & Hüffmeier, J. (2023) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2023.2206571 · Cited by 20
“Given these incentive structures, researchers typically aim to publish their work in (prestigious) journals, and they correctly assume that statistically significant results are more likely to be published than non-significant results (e.g., Fanelli, 2012; GinerSorolla, 2012; Sterling, 1959).”
Background
Conceptual and analytical approaches for modelling the developmental origins of inequality
Malani, A., Archie, E. A., & Rosenbaum, S. (2023) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2022.0306 · Cited by 9
“Assuming the ‘file drawer’ effect applies to this literature [57], then false positives may be over-represented relative to false negatives.”
Background
Accounting for intangible assets – insights from meta-analysis of R&D research
Jeny, A., & Moldovan, R. (2021) · Journal of Accounting Literature · doi:10.1108/jal-11-2021-0004 · Cited by 17
Occurrence of Two-Year Cyclicity, “Saw-Blade Fluctuation”, in Vendace Populations in Finland
Marjomäki, T., Auvinen, H., Helminen, H., Huusko, A., Huuskonen, H., Hyvärinen, P., Jurvelius, J., Sarvala, J., Valkeajärvi, P., Viljanen, M., et al. (2021) · Annales Zoologici Fennici · doi:10.5735/086.058.0408 · Cited by 8
“Thus, spurious correlations are likely to occur in a data set of many independent lake-specific studies and publication bias (Sterling 1959) may have promoted their conspicuousness.”
Background
Steady-State Visually Evoked Potentials and Feature-based Attention: Preregistered Null Results and a Focused Review of Methodological Considerations
Adam, K. C. S., Chang, L., Rangan, N., & Serences, J. T. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01665 · Cited by 13
Clinical trial registration and publication in acupuncture studies: a systematic review
MATSUURA, Y., Welch, N. T., SAKAI, T., & Tsutani, K. (2020) · Integrative Medicine Research · doi:10.1016/j.imr.2020.01.008 · Cited by 11
The fragility of results and bias in empirical research: an exploratory exposition
Moosa, I. A. (2018) · Journal of Economic Methodology · doi:10.1080/1350178x.2018.1556798 · Cited by 17
“They further argued that the conventional econometric methodology ‘generates inference only if a precisely defined model were available, and which can be used to explore the sensitivity of inferences only to discrete changes in assumptions’. According to Leamer and Leonard (1983), this practice ‘deflects econometric t…”
BackgroundInfluential
Accuracy of Effect Size Estimates From Published Psychological Experiments Involving Multiple Trials
Brand, A., Bradley, M. T., Best, L. A., & Stoica, G. (2011) · The Journal of general psychology · doi:10.1080/00221309.2011.604365 · Cited by 8
“The problems of inaccuracy arose through the predominant and longstanding bias to publish results that were statistically significant even when those studies were underpowered (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
Background
Commentary: Toward systematic reviews in epidemiology
Bracken, M. B. (2001) · International Journal of Epidemiology · doi:10.1093/ije/30.5.954 · Cited by 7
Is unreliability in peer review harmful?
Roediger, H. L., III (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065936 · Cited by 8
Evidence of the paranormal: A skeptic's reactions
Gardner, M. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054686 · Cited by 8
A to-do about dualism or a duel about data?
Alcock, J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00054972 · Cited by 8
Seasonal variations in the birth of the mentally deficient?
Sterling, T. (1960) · American Journal of Public Health and the Nations Health · doi:10.2105/ajph.50.7.955 · Cited by 8
Self-citation
Single paper meta‐analysis is unavoidable
McShane, B., & Böckenholt, U. (2025) · Journal of Consumer Psychology · doi:10.1002/jcpy.1462 · Cited by 7
Balanced examination of positive publication bias impact
Ramos, M. (2025) · Accountability in Research · doi:10.1080/08989621.2025.2538066 · Cited by 12
“Sterling (1959) first defined this as the phenomenon where studies that produce statistically significant results are more likely to be published than those that did not.”
Replication Studies Using Secondary or Nonexperimental Datasets
Huang, F. L., & Huang, A. B. (2024) · School Psychology Review · doi:10.1080/2372966x.2024.2346781 · Cited by 8
“Publication bias refers to the selective publication of studies that report statistically significant findings and those studies with nonsignificant results are more likely to be rejected (Sterling, 1959).”
Time-Perception Deficits in Attention-Deficit/Hyperactivity Disorder: A Systematic Review and Meta-Analysis
Metcalfe, K. B., McFeaters, C. D., & Voyer, D. (2023) · Developmental Neuropsychology · doi:10.1080/87565641.2023.2293712 · Cited by 11
“This raises the possibility that our final sample might not be representative of the entire population of studies in existence (Rosenthal, 1979) and research with non-significant results might not have been included in our meta-analysis given their lower likelihood of publication (Sterling, 1959).”
Background

Authors: Sterling, T. D.
Affiliations: University of Cincinnati (US)
Corresponding author: Sterling, Theodore D.
Venue: Journal of the American Statistical Association
Publisher: Informa UK Limited
Published: 1959-03
Volume: 54
Issue: 285
Pages: 30-34
ISSN: 0162-1459 (ISSN-L)
1537-274X
DOI: 10.1080/01621459.1959.10501497
OpenAlex: W2062908954
Cited by: 964
References: 0
Citations by year: 2026: 8 · 2025: 17 · 2024: 18 · 2023: 35 · 2022: 46 · 2021: 48 · 2020: 55 · 2019: 46 · 2018: 51 · 2017: 55 · 2016: 34 · 2015: 31 · 2014: 43 · 2013: 32 · 2012: 30
FWCI: 9.60
Type: journal article
Language: en
Open access: closed
Locations: Journal of the American Statistical Association (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Education › Academic integrity and plagiarism
Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords: publication bias, non-significant results, selective publication, significance testing, false positives, replication, research credibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Metadata

Record

About these checks