The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Abstract
Psychology has recently been viewed as facing a replication crisis because efforts to replicate past study findings frequently do not show the same result. Often, the first study showed a statistically significant result but the replication does not. Questions then arise about whether the first study results were false positives, and whether the replication study correctly indicates that there is truly no effect after all. This article suggests these so-called failures to replicate may not be failures at all, but rather are the result of low statistical power in single replication studies, and the result of failure to appreciate the need for multiple replications in order to have enough power to identify true effects. We provide examples of these power problems and suggest some solutions using Bayesian statistics and meta-analysis. Although the need for multiple replication studies may frustrate those who would prefer quick answers to psychology's alleged crisis, the large sample sizes typically needed to provide firm evidence will almost always require concerted efforts from multiple investigators. As a result, it remains to be seen how many of the recently claimed failures to replicate will be supported or instead may turn out to be artifacts of inadequate sample sizes and single study replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
(2002) · Cited by 13,211
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 2×
“Similarly, we agree that the continuation of underpowered studies in many areas of psychology (e.g., Button et al., 2013) undermines scientific psychology.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Scientists at the biotechnology firm Amgen attempted to replicate 53 landmark studies in hematology and oncology, but were able to confirm the original findings in only six cases, implying an apparent failure rate of 89% (Begley & Ellis, 2012).”
BackgroundFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
“Maxwell, Lau, and Howard (2015) suggest that replication studies demonstrate the absence of an effect by using equivalence bounds of D L ¼ (cid:2) .1 and D U ¼ .1 or even D L ¼ (cid:2) .05 and D U ¼ .05.”
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
Lack of group-to-individual generalizability is a threat to human subjects research
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711978115 · Cited by 1,099
Indices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 1,435
“…of open-sourcing their models/systems when possible and provide sufficient details about the test datasets, which are essential for others to replicate and validate existing innovations across different contexts, preventing the potential pitfall of another replication crisis (Maxwell et al., 2015).”
BackgroundWicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Podobnik, B., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 643
Human handedness: A meta-analysis.
Papadatou-Pastou, M., Ntolka, E., Schmitz, J., Martin, M., Munafò, M. R., Ocklenburg, S., & Paracchini, S. (2020) · Psychological Bulletin · doi:10.1037/bul0000229 · Cited by 526
A systematic review and meta-analysis of psychological interventions to improve mental wellbeing
van Agteren, J., Iasiello, M., Lo, L., Bartholomaeus, J., Kopsaftis, Z., Carey, M., & Kyrios, M. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01093-w · Cited by 510
Positive Affect and Health: What Do We Know and Where Next Should We Go?
Pressman, S. D., Jenkins, B. N., & Moskowitz, J. T. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102955 · Cited by 452
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382
“Third, if the Reproducibility Project: Psychology (Open Science Collaboration, 2015) taught us one thing about the state of the field, it is that psychologists have difficulty agreeing on whether findings have been successfully replicated (Maxwell et al., 2015).”
BackgroundAnderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Self-citationContextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
Altered Brain Activity in Unipolar Depression Revisited: Meta-analyses of Neuroimaging Studies
Müller, V. I., Cieslik, E. C., Serbanescu, I., Laird, A. R., Fox, P. T., & Eickhoff, S. B. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.2783 · Cited by 304
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295
“The small degrees of evidence provided by these published reports, taken at face value, are consistent with observations by Wetzels et al. (2011) as well as the cautionary messages by Johnson (2013) and Maxwell, Lau, and Howard (2015).”
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
Replication in Psychological Science
Lindsay, D. S. (2015) · Psychological Science · doi:10.1177/0956797615616374 · Cited by 280
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
The Nomological Net of the HEXACO Model of Personality: A Large-Scale Meta-Analytic Investigation
Zettler, I., Thielmann, I., Hilbig, B. E., & Moshagen, M. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619895036 · Cited by 257
A Guide to Conducting a Meta-Analysis
Cheung, M. W.-L., & Vijayakumar, R. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9319-z · Cited by 239
Trending Norms: A Lever for Encouraging Behaviors Performed by the Minority
Mortensen, C. R., Neel, R., Cialdini, R. B., Jaeger, C. M., Jacobson, R. P., & Ringel, M. M. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550617734615 · Cited by 221
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167
Self-citationBlake, K. R., & Gangestad, S. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220913363 · Cited by 159
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Howard, M. C., & Henderson, J. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2022.113328 · Cited by 134
Is mindfulness research methodology improving over time? A systematic review
Goldberg, S. B., Tucker, R. P., Greene, P. A., Simpson, T. L., Kearney, D. J., & Davidson, R. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0187298 · Cited by 125
The effects of information and social conformity on opinion change
Mallinson, D. J., & Hatemi, P. K. (2018) · PLOS ONE · doi:10.1371/journal.pone.0196600 · Cited by 62
Tello, N., Harika-Germaneau, G., Serra, W., Jaafari, N., & Chatard, A. (2020) · Psychological Science · doi:10.1177/0956797619893062 · Cited by 54
Questions and answers about conducting systematic reviews in sport and exercise psychology
Gunnell, K., Poitras, V. J., & Tod, D. (2020) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2019.1695141 · Cited by 45
Results Blind Science Publishing
Locascio, J. J. (2017) · Basic and Applied Social Psychology · doi:10.1080/01973533.2017.1336093 · Cited by 45
Hardwicke, T. E., Szűcs, D., Thibault, R. T., Crüwell, S., van den Akker, O. R., Nuijten, M. B., & Ioannidis, J. P. A. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211040837 · Cited by 44
Song, H., Markowitz, D. M., & Taylor, S. H. (2022) · Journal of Communication · doi:10.1093/joc/jqac017 · Cited by 41
Kumpfer, K. L., Scheier, L. M., & Brown, J. (2020) · Evaluation & the Health Professions · doi:10.1177/0163278718772886 · Cited by 41
Context Moderates Affirmation Effects on the Ethnic Achievement Gap
Protzko, J., & Aronson, J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616646426 · Cited by 39
Chen, Z., Hu, B., Liu, X., Becker, B., Eickhoff, S. B., Miao, K., Gu, X., Tang, Y., Dai, X., Li, C., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-02941-4 · Cited by 35
Are people with social anxiety disorder happier alone?
Goodman, F. R., Rum, R., Silva, G., & Kashdan, T. B. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102474 · Cited by 33
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32
Treger, S., Benau, E. M., & Timko, C. A. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285267 · Cited by 25
“In closely examining why a study failed to replicate a previously established phenomenon, it is important to evaluate the stimuli, method of administration, and sample characteristics [87].”
BackgroundAllegiance effects in mindfulness-based interventions for psychiatric disorders: A meta-re-analysis
Goldberg, S. B., & Tucker, R. P. (2020) · Psychotherapy Research · doi:10.1080/10503307.2019.1664783 · Cited by 21
The One Ring Model: Rape Culture Beliefs are Linked to Purity Culture Beliefs
Klement, K. R., Sagarin, B. J., & Skowronski, J. J. (2022) · Sexuality & Culture · doi:10.1007/s12119-022-09986-2 · Cited by 18
Giurge, L. M., Lin, E. H.-L., & Effron, D. A. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104144 · Cited by 3
Flynn, M. K., & Hebert, E. R. (2022) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2021.11.005 · Cited by 2
Metadata
- Authors
- Maxwell, S. E., Lau, M. Y., & Howard, G. S.
- Venue
- American Psychologist
- Published
- 2015
- Volume
- 70
- Issue
- 6
- Pages
- 487-498
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W1470195681
- Cited by
- 1,005
- References
- 65
- FWCI
- 62.49
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.