What meta-analyses reveal about the replicability of psychological research.
Open accessAbstract
Can recent failures to replicate psychological research be explained by typical magnitudes of statistical power, bias or heterogeneity? A large survey of 12,065 estimated effect sizes from 200 meta-analyses and nearly 8,000 papers is used to assess these key dimensions of replicability. First, our survey finds that psychological research is, on average, afflicted with low statistical power. The median of median power across these 200 areas of research is about 36%, and only about 8% of studies have adequate power (using Cohen's 80% convention). Second, the median proportion of the observed variation among reported effect sizes attributed to heterogeneity is 74% (I2). Heterogeneity of this magnitude makes it unlikely that the typical psychological study can be closely replicated when replication is defined as study-level null hypothesis significance testing. Third, the good news is that we find only a small amount of average residual reporting bias, allaying some of the often-expressed concerns about the reach of publication bias and questionable research practices. Nonetheless, the low power and high heterogeneity that our survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology. (PsycINFO Database Record (c) 2018 APA, all rights reserved).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046 · see citing works
Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis.
Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014) · Psychological Bulletin · doi:10.1037/a0035661 · Cited by 1,981 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247 · see citing works
Appealing to fear: A meta-analysis of fear appeal effectiveness and theories
Tannenbaum, M. B., Hepler, J., Zimmerman, R. S., Saul, L., Jacobs, S., Wilson, K., & Albarracín, D. (2015) · Psychological Bulletin · doi:10.1037/a0039729 · Cited by 1,203 · see citing works
Ingroup favoritism in cooperation: a meta-analysis
Balliet, D., Wu, J., & De Dreu, C. K. W. (2014) · Psychological Bulletin · doi:10.1037/a0037737 · Cited by 1,126 · see citing works
Fischer, P., Krueger, J. I., Greitemeyer, T., Vogrincic, C., Kastenmüller, A., Frey, D., Heene, M., Wicher, M., & Kainbacher, M. (2011) · Psychological Bulletin · doi:10.1037/a0023304 · Cited by 983 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
Self-citationThe psychological effects of meditation: a meta-analysis
Sedlmeier, P., Eberth, J., Schwarz, M., Zimmermann, D., Haarig, F., Jaeger, S., & Kunze, S. (2012) · Psychological Bulletin · doi:10.1037/a0028168 · Cited by 910 · see citing works
Shame, guilt, and depressive symptoms: A meta-analytic review.
Kim, S., Thibodeau, R., & Jorgensen, R. S. (2011) · Psychological Bulletin · doi:10.1037/a0021466 · Cited by 866 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Heritability of personality: A meta-analysis of behavior genetic studies.
Vukasović, T., & Bratko, D. (2015) · Psychological Bulletin · doi:10.1037/bul0000017 · Cited by 716 · see citing works
Verhage, M. L., Schuengel, C., Madigan, S., Fearon, R. M. P., Oosterman, M., Cassibba, R., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2016) · Psychological Bulletin · doi:10.1037/bul0000038 · Cited by 550 · see citing works
Brewin, C. R. (2014) · Psychological Bulletin · doi:10.1037/a0033722 · Cited by 492 · see citing works
Compensatory control and the appeal of a structured world.
Landau, M. J., Kay, A. C., & Whitson, J. A. (2015) · Psychological Bulletin · doi:10.1037/a0038703 · Cited by 479 · see citing works
Physiological concomitants of perseverative cognition: A systematic review and meta-analysis.
Ottaviani, C., Thayer, J. F., Verkuil, B., Lonigro, A., Medea, B., Couyoumdjian, A., & Brosschot, J. F. (2016) · Psychological Bulletin · doi:10.1037/bul0000036 · Cited by 476 · see citing works
Gender differences in self-conscious emotional experience: A meta-analysis.
Else-Quest, N. M., Higgins, A., Allison, C., & Morton, L. C. (2012) · Psychological Bulletin · doi:10.1037/a0027930 · Cited by 464 · see citing works
Pigott, T. D. (2012) · doi:10.1007/978-1-4614-2278-5 · Cited by 453 · see citing works
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409 · see citing works
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · see citing works
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
Williams, M. J., & Tiedens, L. Z. (2016) · Psychological Bulletin · doi:10.1037/bul0000039 · Cited by 374 · see citing works
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373 · see citing works
Self-citationMeta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364 · see citing works
Sex differences in cooperation: A meta-analytic review of social dilemmas.
Balliet, D., Li, N. P., Macfarlan, S. J., & Van Vugt, M. (2011) · Psychological Bulletin · doi:10.1037/a0025354 · Cited by 349 · see citing works
Johnsen, T. J., & Friborg, O. (2015) · Psychological Bulletin · doi:10.1037/bul0000015 · Cited by 331 · see citing works
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · see citing works
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313 · see citing works
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288 · see citing works
Lui, P. P. (2015) · Psychological Bulletin · doi:10.1037/a0038449 · Cited by 285 · see citing works
Limitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229 · see citing works
Self-citationRegistered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223 · see citing works
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219 · see citing works
Neither fixed nor random: weighted least squares meta‐regression
Stanley, T. D., & Doucouliagos, H. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1211 · Cited by 205 · see citing works
Self-citationKredlow, M. A., Unger, L. D., & Otto, M. W. (2016) · Psychological Bulletin · doi:10.1037/bul0000034 · Cited by 130 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
Self-citationHow many words do we read per minute? A review and meta-analysis of reading rate
Brysbaert, M. (2019) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104047 · Cited by 553 · see citing works
Sermondade, N., Huberlant, S., Bourhis-Lefebvre, V., Arbo, E., Gallot, V., Colombani, M., & Fréour, T. (2019) · Human Reproduction Update · doi:10.1093/humupd/dmz011 · Cited by 397 · see citing works
Testing for funnel plot asymmetry of standardized mean differences
Pustejovsky, J. E., & Rodgers, M. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1332 · Cited by 354 · see citing works
Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence
Appel, M., Marker, C., & Gnambs, T. (2020) · Review of General Psychology · doi:10.1177/1089268019880891 · Cited by 290 · see citing works
Pizarro, J. J., Zumeta, L. N., Bouchat, P., Włodarczyk, A., Rimé, B., Basabe, N., Amutio, A., & Páez, D. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.974683 · Cited by 88 · see citing works
Calculating statistical power for meta-analysis using metapower
Griffin, J. W. (2021) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.17.1.p024 · Cited by 84 · see citing works
The sleep benefit in episodic memory: An integrative review and a meta-analysis.
Berres, S., & Erdfelder, E. (2021) · Psychological Bulletin · doi:10.1037/bul0000350 · Cited by 74 · see citing works
Meta-analyses indexed in PsycINFO had a better completeness of reporting when they mention PRISMA
Leclercq, V., Beaudart, C., Ajamieh, S., Rabenda, V., Tirelli, E., & Bruyère, O. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.014 · Cited by 69 · see citing works
Davidson, P., Jönsson, P., Carlsson, I., & Pace-Schott, E. (2021) · Nature and Science of Sleep · doi:10.2147/nss.s286701 · Cited by 59 · see citing works
A Meta-analytic Review of the Relationship Between Posttraumatic Growth, Anxiety, and Depression
Long, L. J., Phillips, C. A., Glover, N., Richardson, A. L., D’Souza, J. M., Cunningham-Erdogdu, P., & Gallagher, M. W. (2021) · Journal of Happiness Studies · doi:10.1007/s10902-021-00370-9 · Cited by 44 · see citing works
Bullock, G. S., Ward, P., Impellizzeri, F. M., Kluzek, S., Hughes, T., Hillman, C., Waterman, B. R., Danelson, K., Henry, K., Barr, E., et al. (2023) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2023.12016 · Cited by 15 · see citing works
Spence, R., & DeMarco, J. (2025) · Behavioral Sciences · doi:10.3390/bs15040487 · Cited by 6 · see citing works
Chen, L., Benjamin, R., Guo, Y., Lai, A., & Heine, S. J. (2023) · Research Square · doi:10.21203/rs.3.rs-1254756/v1 · Cited by 5 · see citing works
- Authors
- Stanley, T. D., Carter, E. C., & Doucouliagos, H.
- Affiliations
- Deakin University (AU)
- DEVCOM Army Research Laboratory (US)
- Venue
- Psychological Bulletin
- Published
- 2018-12
- Volume
- 144
- Issue
- 12
- Pages
- 1325-1346
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2896198545
- Cited by
- 307
- References
- 48
- Citations by year
- 2026: 16 · 2025: 28 · 2024: 33 · 2023: 50 · 2022: 43 · 2021: 54 · 2020: 49 · 2019: 27 · 2018: 7
- FWCI
- 26.12
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-06-13)
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- replicability, replication failure, statistical power, publication bias, reporting bias, heterogeneity, effect size, meta-analysis, questionable research practices, null hypothesis significance testing
- MeSH terms
- Data Interpretation, Statistical, Humans, Psychology, Research Design, Meta-Analysis as Topic, Reproducibility of Results, Publication Bias, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
