The DOI matches its Crossref record; 64 of the 65 references it cites were found in a registry or an index.
Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size
Open accessAbstract
BACKGROUND: The p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias. METHODS: We investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values. RESULTS: We found a negative correlation of r = -.45 [95% CI: -.53; -.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings. CONCLUSION: The negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The correlation between sample size and ES was estimated to be as high as -0.52 in tests of mean difference,40 indicating that studies with larger samples produce smaller ESs.”
From Psychological interventions for depression and anxiety: a systematic review and meta-analysis of Iranian chronic pain trials. (2020) · cites this paper as [40]
In the citing paper
“Because many effects in the biomedical, behavioural and social sciences are small [60–62], it is important for studies to be sufficiently high-powered.”
From The natural selection of bad science (2016) · cites this paper as [62]
In the citing paper
“As an impact of a high risk of bias11 and a low sample size12 of the studies included into meta-analyses is known, these potential confounders should be considered in dose-response-analyses, likewise.”
From Dose-response-relationship of stabilisation exercises in patients with chronic non-specific low back pain: a systematic review with meta-regression (2020) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Consequently, a substantial part of what we think is secure knowledge might actually be (statistical) errors [46–49].”
BackgroundPublication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
(2011) · Cited by 2,025
On the Use of Beta Coefficients in Meta-Analysis.
Peterson, R. A., & Brown, S. P. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.1.175 · Cited by 1,643
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Cited by
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
“Because such p-values require large observed effect sizes in small designs, the effects sizes reported in the literature tend to be inflated, also those from studies that can be replicated (Kuhberger et al., 2014; Open Science Collaboration, 2015; Vasishth & Gelman, 2017).”
ResultThe natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Yarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701
Metadata
- Authors
- Kühberger, A., Fritz, A., & Scherndl, T.
- Affiliations
- University of Salzburg (AT)
- Corresponding author
- Kühberger, Anton
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-09-05
- Volume
- 9
- Issue
- 9
- Pages
- e105825
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4156299
- OpenAlex
- W1998410565
- Cited by
- 399
- References
- 65
- Citations by year
- 2026: 10 · 2025: 36 · 2024: 27 · 2023: 44 · 2022: 45 · 2021: 44 · 2020: 38 · 2019: 35 · 2018: 35 · 2017: 29 · 2016: 23 · 2015: 30 · 2014: 2
- FWCI
- 27.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Österreichischen Akademie der Wissenschaften
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, effect size, sample size, p-values, statistical significance, power analysis, negative correlation, p-value distribution, psychology research
- MeSH terms
- Mental Disorders, Humans, Psychology, Research, Epidemiologic Research Design, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. https://app.cochat.ai/papers/10.1371/journal.pone.0105825
- The paper
- Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PLoS ONE. https://doi.org/10.1371/journal.pone.0105825
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.