Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite methodological challenges10–11, many of the findings of cognitive neuroscience provide a solid basis on which to build.”
From Cognitive computational neuroscience (2018) · cites this paper as [10]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13 -16).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“such studies showed that power varies widely between subfields or methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Szucs and Ioannidis, 2017
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 9×
“…examined p value reporting errors in the cognitive neuroscience literature as recent studies found that 12.5% to 20% of psychology papers have major p value reporting errors which affect judgments about statistical significance (Bakker and Wicherts, 2011; Nuijten et al. 2015; Veldkamp et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationCluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,327
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Atkinson, A., Barnett, V., & Lewis, T. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983451 · Cited by 1,682
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Self-citationPuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475
A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures
De Filippis, F., La Storia, A., Stellato, G., Gatti, M., & Ercolini, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089680 · Cited by 1,398
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Balakrishnan, N., Barnett, V., & Lewis, T. (1995) · Biometrics · doi:10.2307/2533352 · Cited by 1,217
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
The Seductive Allure of Neuroscience Explanations
Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20040 · Cited by 1,063
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Calibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 854
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Self-citationThe Empire of Chance: How Probability Changed Science and Everyday Life
Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., & Kruger, L. (1989) · Cambridge University Press eBooks · doi:10.1017/cbo9780511720482 · Cited by 754
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Self-citationDo studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
The Empire of chance: how probability changed science and everyday life
(1989) · Choice Reviews Online · doi:10.5860/choice.27-2084 · Cited by 597
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564
(1990) · Cited by 561
Yarkoni, T. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01127.x · Cited by 549
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
The secret lives of experiments: Methods reporting in the fMRI literature
Carp, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.004 · Cited by 457
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Barnett, V., & Lewis, T. (1974) · Cited by 445
Evolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
Self-citationThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323
A survey of the statistical power of research in behavioral ecology and animal behavior
Jennions, M. D. (2003) · Behavioral Ecology · doi:10.1093/beheco/14.3.438 · Cited by 278
Surrogate Science: The Idol of a Universal Method for Scientific Inference
Gigerenzer, G., & Marewski, J. N. (2015) · Journal of Management · doi:10.1177/0149206314547522 · Cited by 215
Bakker, M., & Wicherts, J. M. (2014) · Psychological Methods · doi:10.1037/met0000014 · Cited by 209
On the probability of making Type I errors.
Pollard, P., & Richardson, J. T. E. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.1.159 · Cited by 172
On the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 152
A Unique Mutation in a MYB Gene Cosegregates with the Nectarine Phenotype in Peach
Vendramin, E., Pea, G., Dondini, L., Pacheco, I., Dettori, M. T., Gazza, L., Scalabrin, S., Strozzi, F., Tartarini, S., Bassi, D., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090574 · Cited by 127
Does parametric fMRI analysis with SPM yield valid results?—An empirical study of 1484 rest datasets
Eklund, A., Andersson, M., Josephson, C., Johannesson, M., & Knutsson, H. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.093 · Cited by 120
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k4bgq · Cited by 114
Statistical power: Concepts, procedures, and applications
Hallahan, M., & Rosenthal, R. (1996) · Behaviour Research and Therapy · doi:10.1016/0005-7967(95)00082-8 · Cited by 100
Distributions of p -values smaller than .05 in psychology: what is going on?
Hartgerink, C. H. J., van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · PeerJ · doi:10.7717/peerj.1935 · Cited by 97
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 83
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69
A Tutorial on Hunting Statistical Significance by Chasing N
Szűcs, D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01444 · Cited by 53
Evolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · Harvard Dataverse · doi:10.7910/dvn/6fmtt3 · Cited by 52
Self-citation(2014) · Data Archiving and Networked Services (DANS) · Cited by 46
Magnitude of effects in clinical trials published in high-impact general medical journals
Siontis, K., Εvangelou, Ε., & Ioannidis, J. P. A. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr095 · Cited by 44
Self-citationSample Size and Power Calculations using the Noncentral t-distribution
Harrison, D. A., & Brady, T. (2004) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0400400205 · Cited by 32
Reliability in Cognitive Neuroscience: A Meta-Meta-Analysis
Uttal, W. R. (2012) · Cited by 24
Reliability in Cognitive Neuroscience: A Meta-Meta Analysis
Klein, C. (2013) · Philosophical Psychology · doi:10.1080/09515089.2013.838818 · Cited by 18
Cited by
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationReproducible brain-wide association studies require thousands of individuals
Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Hendrickson, T. J., et al. (2022) · Nature · doi:10.1038/s41586-022-04492-9 · Cited by 1,928
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
“…on being true for many studies run today (Cohen, 1962; Dumas-Mallet, Button, Boraud, Gonon, & Munafò, 2017; Fraley & Vazire, 2014; Maxwell, 2004; Szucs & Ioannidis, 2017a; Vankov, Bowers, & Munafo, 2014), despite the fact that many articles on power calculation and required sample sizes have…”
BackgroundIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“Analysis of the literature suggests that publications are indeed too often on the edge of significance (Szucs and Ioannidis, 2016) and are vastly biased by selection according to the prevailing opinions (Ioannidis, 2008).”
Background
Metadata
- Authors
- Szucs, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-02
- Volume
- 15
- Issue
- 3
- Pages
- e2000797
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5333800
- OpenAlex
- W2951081425
- Cited by
- 761
- References
- 62
- Citations by year
- 2026: 28 · 2025: 43 · 2024: 120 · 2023: 75 · 2022: 80 · 2021: 76 · 2020: 94 · 2019: 102 · 2018: 91 · 2017: 46
- FWCI
- 81.43
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-03-05)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo (cc-by)
- Funders
- James S. McDonnell Foundation (220020370)
- Universiteit van Tilburg
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- effect size, statistical power, cognitive neuroscience, psychology literature, sample size, replication success, journal impact factor, null hypothesis, publication bias, statistical records
- MeSH terms
- Cognitive Neuroscience, Humans, Probability, Psychology, Publications, Surveys and Questionnaires, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.