Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size
Open accessThe DOI matches its Crossref record; 64 of the 65 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias. METHODS: We investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values. RESULTS: We found a negative correlation of r = -.45 [95% CI: -.53; -.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings. CONCLUSION: The negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The correlation between sample size and ES was estimated to be as high as -0.52 in tests of mean difference,40 indicating that studies with larger samples produce smaller ESs.”
From Psychological interventions for depression and anxiety: a systematic review and meta-analysis of Iranian chronic pain trials. (2020) · cites this paper as [40]
In the citing paper
“Because many effects in the biomedical, behavioural and social sciences are small [60–62], it is important for studies to be sufficiently high-powered.”
From The natural selection of bad science (2016) · cites this paper as [62]
In the citing paper
“As an impact of a high risk of bias11 and a low sample size12 of the studies included into meta-analyses is known, these potential confounders should be considered in dose-response-analyses, likewise.”
From Dose-response-relationship of stabilisation exercises in patients with chronic non-specific low back pain: a systematic review with meta-regression (2020) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Consequently, a substantial part of what we think is secure knowledge might actually be (statistical) errors [46–49].”
BackgroundPublication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
(2011) · Cited by 2,025
On the Use of Beta Coefficients in Meta-Analysis.
Peterson, R. A., & Brown, S. P. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.175 · Cited by 1,643 · cited 1×
“ES for regression analysis were converted according to Peterson and Brown [37].”
MethodContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.94 · Cited by 1,216 · cited 1×
“This is plausible, but not fully consistent with the long known problem that moderation effects are studied with strongly underpowered samples and evidence that sample size of moderation studies are not much larger than ‘typical’ sample size [14].”
BackgroundNull hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“Publication bias can occur at any stage of the publication process where a decision is made [18,19]: in the researcher’s decision to write up a manuscript [20]; in the decision to submit the manuscript to a journal [21,22]); in the decision of journal editors to send a paper out for review [23]; in the reviewer’s reco…”
BackgroundPublication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · cited 1×
“However, as Stern and Simes [45] pointed out, individual factors cannot be assumed to be independent from editorial factors, since previous experiences may have conditioned authors to expect rejection of non-significant studies [20].”
BackgroundThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 1×
“Fourth, it has been argued that authors could act strategically and set up smaller experiments in order to have more maneuverability when analyzing a set of experiments [15].”
BackgroundUnderstanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis
Cumming, G. (2013) · Routledge eBooks · doi:10.4324/9780203807002 · Cited by 790
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 1×
“Most of the reported examples for the decline effect stem from sciences other than psychology, but Fanelli [52] found that results giving support to the research hypothesis increase down the hierarchy of the sciences.”
ResultGermine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · cited 1×
“Yet, empirical evidence does not fully support this view as there are strikingly similar results from MTurk as well as lab-experiments [13].”
ResultFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 3×
“There are many degrees of freedom for conducting studies and analyzing results that increase the probability of getting a significant result [26], and may therefore be used in order to minimize the danger of non-significant results [27].”
BackgroundSample size calculations in randomised trials: mandatory and mystical
Schulz, K. F., & Grimes, D. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)61034-3 · Cited by 549
(2006) · Educational Researcher · doi:10.3102/0013189x035006033 · Cited by 543
Ferguson, C. J., & Brannick, M. T. (2012) · Psychological Methods · doi:10.1037/a0024445 · Cited by 493
Cognition as Intuitive Statistics
Gigerenzer, G., & Murray, D. J. (2015) · Psychology Press eBooks · doi:10.4324/9781315668796 · Cited by 453
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 1×
“Data sharing practices have been found to be somewhat lacking and have been put forward as one reason impeding scientific progress and replication [41].”
BackgroundTesting and adjusting for publication bias
Møller, A. P., & Jennions, M. D. (2001) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(01)02235-2 · Cited by 418
Thompson, B. (2002) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2002.tb00167.x · Cited by 398
The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education
Slavin, R., & Smith, D. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709352369 · Cited by 360
Unpublished results hide the decline effect
Schooler, J. (2011) · Nature · doi:10.1038/470437a · Cited by 333
A Meta-Analysis Summarizing the Effects of Pornography II Aggression After Exposure
ALLEN, M., D'ALESSIO, D., & BREZGEL, K. (1995) · Human Communication Research · doi:10.1111/j.1468-2958.1995.tb00368.x · Cited by 310
Effects of media violence on viewers' aggression in unconstrained social interaction.
Wood, W., Wong, F. Y., & Chachere, J. G. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.109.3.371 · Cited by 303
Gerber, A., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 284
Reporting of sample size calculation in randomised controlled trials: review
Charles, P., Giraudeau, B., Dechartres, A., Baron, G., & Ravaud, P. (2009) · BMJ · doi:10.1136/bmj.b1732 · Cited by 274
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · cited 1×
“Publication bias has been shown in a diverse range of research areas (political science [29], sociology [39,40], evolutionary biology [42] and also in some areas of psychology [43,44]) However, most of these prevalence estimates have been based on the analysis of some journal volumes over a specific course of time or…”
BackgroundGerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 246
Could It Be Better to Discard 90% of the Data? A Statistical Paradox
Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010) · The American Statistician · doi:10.1198/tast.2009.08205 · Cited by 170
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.
Coursol, A., & Wagner, E. E. (1986) · Professional Psychology: Research and Practice · doi:10.1037//0735-7028.17.2.136 · Cited by 165
Testing for Publication Bias in Political Science
Gerber, A. S., Green, D. P., & Nickerson, D. (2001) · Political Analysis · doi:10.1093/oxfordjournals.pan.a004877 · Cited by 157
Systematic review of publication bias in studies on publication bias
Dubben, H.-H., & Beck-Bornholdt, H.-P. (2005) · BMJ · doi:10.1136/bmj.38478.497164.f7 · Cited by 153
Schneider, B. (2006) · Cited by 149
Finding the missing science: The fate of studies submitted for review by a human subjects committee.
Cooper, H., DeNeve, K., & Charlton, K. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.4.447 · Cited by 144 · cited 1×
“Publication bias can occur at any stage of the publication process where a decision is made [18,19]: in the researcher’s decision to write up a manuscript [20]; in the decision to submit the manuscript to a journal [21,22]); in the decision of journal editors to send a paper out for review [23]; in the reviewer’s reco…”
BackgroundA Critical Assessment of Null Hypothesis Significance Testing in Quantitative Communication Research
Levine, T. R., Weber, R., Hullett, C., Park, H. S., & Lindsey, L. L. M. (2008) · Human Communication Research · doi:10.1111/j.1468-2958.2008.00317.x · Cited by 138
Problems With Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say?
Gliner, J. A., Leech, N. L., & Morgan, G. A. (2002) · The Journal of Experimental Education · doi:10.1080/00220970209602058 · Cited by 136
The frequency of excess success for articles in Psychological Science
Francis, G. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0601-x · Cited by 135
Levine, T. R., Asada, K. J., & Carpenter, C. (2009) · Communication Monographs · doi:10.1080/03637750903074685 · Cited by 131
US studies may overestimate effect sizes in softer research
Fanelli, D., & Ioannidis, J. P. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1302997110 · Cited by 130
Atkinson, D. R., Furlong, M. J., & Wampold, B. E. (1982) · Journal of Counseling Psychology · doi:10.1037/0022-0167.29.2.189 · Cited by 109
Hate Speech and Academic Freedom in the Academy
Tierney, W. G. (2006) · Educational Researcher · doi:10.3102/0013189x035003033 · Cited by 88
Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests
Lecoutre, M., Poitevineau, J., & Lecoutre, B. (2003) · International Journal of Psychology · doi:10.1080/00207590244000250 · Cited by 86 · cited 1×
“[11] showed that researchers collect more data if the results are nearly, but not quite, statistically significant, and John et al.”
BackgroundConfidence Intervals: Better Answers to Better Questions
Cumming, G., & Fidler, F. (2009) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.217.1.15 · Cited by 83 · cited 1×
“The reader can tell from the width of the intervals how accurate, and therefore trustworthy, the estimation is [60].”
BackgroundWhy animal research needs to improve
Macleod, M. R. (2011) · Nature · doi:10.1038/477511a · Cited by 83
Fritz, A., Scherndl, T., & Kühberger, A. (2012) · Theory & Psychology · doi:10.1177/0959354312436870 · Cited by 80
Self-citationSilva-Ayçaguer, L. C., Suárez-Gil, P., & Fernández-Somoano, A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-44 · Cited by 67
The Life of p : “Just Significant” Results are on the Rise
Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.863371 · Cited by 64
Odgaard, E. C., & Fowler, R. L. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0019294 · Cited by 60
Publication practices and the file drawer problem: A survey of published authors.
Rotton, J., Foos, P. W., Meek, L. V., & Levitt, M. J. (1995) · Journal of social behavior and personality · Cited by 41
France, B. H. L., Heisel, A. D., & Beatty, M. J. (2004) · Communication Monographs · doi:10.1080/03634520410001693148 · Cited by 38
Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C. J., Hoey, J., Horton, R., Kotzin, S., Laine, C. M., Marušić, A., Overbeke, A. J. P. M., et al. (2004) · PubMed · Cited by 37
RIDLEY, J., KOLM, N., FRECKELTON, R. P., & GAGE, M. J. G. (2007) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2006.01291.x · Cited by 34
What's Wrong with Research Literatures? And how to Make Them Right
Howard, G. S., Hill, T. L., Maxwell, S. E., Baptista, T. M., Farias, M. H., Coelho, C., Coulter-Kern, M., & Coulter-Kern, R. (2009) · Review of General Psychology · doi:10.1037/a0015319 · Cited by 22
A Brief Primer on Effect Sizes
Thompson, B. (2009) · Journal of Teaching in Physical Education · doi:10.1123/jtpe.28.3.251 · Cited by 17
Publication of Nonsignificant Results: A Survey of Psychologists' Opinions
Reysen, S. (2006) · Psychological Reports · doi:10.2466/pr0.98.1.169-175 · Cited by 17
Accuracy of Effect Size Estimates From Published Psychological Experiments Involving Multiple Trials
Brand, A., Bradley, M. T., Best, L. A., & Stoica, G. (2011) · The Journal of General Psychology · doi:10.1080/00221309.2011.604365 · Cited by 10
Methods: why are effect sizes still neglected?
Morris, P. E., & Fritz, C. O. (2013) · Cited by 2
Cited by
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
“Because such p-values require large observed effect sizes in small designs, the effects sizes reported in the literature tend to be inflated, also those from studies that can be replicated (Kuhberger et al., 2014; Open Science Collaboration, 2015; Vasishth & Gelman, 2017).”
ResultThe natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Yarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Metadata
- Authors
- Kühberger, A., Fritz, A., & Scherndl, T.
- Affiliations
- University of Salzburg (AT)
- Corresponding author
- Kühberger, Anton
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-09-05
- Volume
- 9
- Issue
- 9
- Pages
- e105825
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4156299
- OpenAlex
- W1998410565
- Cited by
- 399
- References
- 65
- Citations by year
- 2026: 10 · 2025: 36 · 2024: 27 · 2023: 44 · 2022: 45 · 2021: 44 · 2020: 38 · 2019: 35 · 2018: 35 · 2017: 29 · 2016: 23 · 2015: 30 · 2014: 2
- FWCI
- 27.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Österreichischen Akademie der Wissenschaften
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, effect size, sample size, p-values, statistical significance, power analysis, negative correlation, p-value distribution, psychology research
- MeSH terms
- Mental Disorders, Humans, Psychology, Research, Epidemiologic Research Design, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.