The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Abstract
Psychology has recently been viewed as facing a replication crisis because efforts to replicate past study findings frequently do not show the same result. Often, the first study showed a statistically significant result but the replication does not. Questions then arise about whether the first study results were false positives, and whether the replication study correctly indicates that there is truly no effect after all. This article suggests these so-called failures to replicate may not be failures at all, but rather are the result of low statistical power in single replication studies, and the result of failure to appreciate the need for multiple replications in order to have enough power to identify true effects. We provide examples of these power problems and suggest some solutions using Bayesian statistics and meta-analysis. Although the need for multiple replication studies may frustrate those who would prefer quick answers to psychology's alleged crisis, the large sample sizes typically needed to provide firm evidence will almost always require concerted efforts from multiple investigators. As a result, it remains to be seen how many of the recently claimed failures to replicate will be supported or instead may turn out to be artifacts of inadequate sample sizes and single study replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
(2002) · Cited by 13,211
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 2×
“Similarly, we agree that the continuation of underpowered studies in many areas of psychology (e.g., Button et al., 2013) undermines scientific psychology.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Scientists at the biotechnology firm Amgen attempted to replicate 53 landmark studies in hematology and oncology, but were able to confirm the original findings in only six cases, implying an apparent failure rate of 89% (Begley & Ellis, 2012).”
BackgroundFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
Diciccio, T. J., Kass, R. E., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,943
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 1×
“It is tempting to infer that the studies have produced truly different results and to begin the search for possible moderators, but as Gelman and Stern (2006) have pointed out, such an inference is often misguided.”
ResultBeyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 1×
“Such concerns led Pashler and Wagenmakers (2012) to conclude that there appears to be a “crisis of confidence in psychological science reflecting an unprecedented doubt among practitioners about the reliability of research findings in the field” (p. 528).”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140 · cited 1×
“In addition, cumulative meta-analysis (Lau et al., 1992) can provide a valuable model by incorporating the information provided in the latest replication study into a previous meta-analysis conducted on prior replication studies.”
MethodConsequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“Cook, Gruder, Henningan, and Faly (1979), Greenwald (1975), and Wilson and Shadish (2006) all offer valuable perspectives on broader issues involved in evaluating the veracity of null effects.”
BackgroundThe Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
Sample Size Calculations in Clinical Research
Chow, S.-C., Wang, H., & Shao, J. (2003) · CRC Press eBooks · doi:10.1201/9780203911341 · Cited by 807
Verbal overshadowing of visual memories: Some things are better left unsaid
Schooler, J. W., & Engstler-Schooler, T. Y. (1990) · Cognitive Psychology · doi:10.1016/0010-0285(90)90003-m · Cited by 794
A tutorial on a practical Bayesian alternative to null-hypothesis significance testing
Masson, M. E. J. (2011) · Behavior Research Methods · doi:10.3758/s13428-010-0049-5 · Cited by 752
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 1×
“More generally, out of 14 replication attempts organized by Nosek and Lakens (2014), nine were interpreted as failing to replicate the original study and the other five were viewed as only partial replications.”
BackgroundThe Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 654
Self-citationThe Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Using significance tests to evaluate equivalence between two experimental groups.
Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.113.3.553 · Cited by 566
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 468
Posterior odds ratios for selected regression hypotheses
Zellner, A., & Siow, A. (1980) · Trabajos de Estadistica Y de Investigacion Operativa · doi:10.1007/bf02888369 · Cited by 463
Keeping One's Distance: The Influence of Spatial Distance Cues on Affect and Evaluation
Williams, L. E., & Bargh, J. A. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02084.x · Cited by 458
How hard is hard science, how soft is soft science? The empirical cumulativeness of research.
Hedges, L. V. (1987) · American Psychologist · doi:10.1037//0003-066x.42.5.443 · Cited by 435
Most published research findings are false-but a little replication goes a long way.
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040028 · Cited by 341
Priming, Replication, and the Hardest Science
Cesario, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513470 · Cited by 320
Assessing uncertainty in physical constants
Henrion, M., & Fischhoff, B. (1986) · American Journal of Physics · doi:10.1119/1.14447 · Cited by 316
The Desperate Need for Replications
Hunter, J. E. (2001) · Journal of Consumer Research · doi:10.1086/321953 · Cited by 280
Effect sizes and p values: What should be reported and what should be replicated?
Greenwald, A., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996) · Psychophysiology · doi:10.1111/j.1469-8986.1996.tb02121.x · Cited by 274
The substitutability of physical and social warmth in daily life.
Bargh, J. A., & Shalev, I. (2012) · Emotion · doi:10.1037/a0023527 · Cited by 246
Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868313507536 · Cited by 245
Registered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223
Rationality in psychological research: The good-enough principle.
Serlin, R. C., & Lapsley, D. K. (1985) · American Psychologist · doi:10.1037//0003-066x.40.1.73 · Cited by 217
Bayesian Versus Frequentist Inference
Wagenmakers, E.-J., Lee, M., Lodewyckx, T., & Iverson, G. J. (2008) · Bayesian Evaluation of Informative Hypotheses · doi:10.1007/978-0-387-09612-4_9 · Cited by 205
Ritchie, S. J., Wiseman, R., & French, C. C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033423 · Cited by 186
An Introduction to Registered Replication Reports at Perspectives on Psychological Science
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614543974 · Cited by 183
The Counternull Value of an Effect Size: A New Statistic
Rosenthal, R., & Rubin, D. B. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00281.x · Cited by 168
Estimating effect size: Bias resulting from the significance criterion in editorial decisions
Lane, D. M., & Dunlap, W. P. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00578.x · Cited by 164
Priming of Social Distance? Failure to Replicate Effects on Social and Food Judgments
Pashler, H., Coburn, N., & Harris, C. R. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042510 · Cited by 164
Correcting the past: Failures to replicate psi.
Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029709 · Cited by 164
On the Post Hoc Power in Testing Mean Differences
Yuan, K.-H., & Maxwell, S. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030002141 · Cited by 160
Self-citationEquivalence confidence intervals for two-group comparisons of means.
Seaman, M. A., & Serlin, R. C. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.403 · Cited by 142
Bonett, D. G. (2012) · Current Directions in Psychological Science · doi:10.1177/0963721412459512 · Cited by 118
History of the sleeper effect: Some logical pitfalls in accepting the null hypothesis.
Cook, T. D., Gruder, C. L., Hennigan, K. M., & Flay, B. R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.4.662 · Cited by 101
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101
You Cannot Step Into the Same River Twice: When Power Analyses Are Optimistic
McShane, B. B., & Böckenholt, U. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614548513 · Cited by 96
Reporting of Statistical Inference in the Journal of Applied Psychology: Little Evidence of Reform
Finch, S. J., Cumming, G., & Thomason, N. R. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971167 · Cited by 86
Pitchforth, J., & Mengersen, K. L. (2012) · Wiley series in probability and statistics · doi:10.1002/9781118394472.ch7 · Cited by 75
Bias in linear model power and sample size calculation due to estimating noncentrality
Taylor, D. J., & Muller, K. E. (1996) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929608831787 · Cited by 49
The perils with the misuse of predictive power
Dallow, N. S., & Fina, P. (2010) · Pharmaceutical Statistics · doi:10.1002/pst.467 · Cited by 34
Wortman, J., Donnellan, M. B., & Lucas, R. E. (2014) · Archives of Scientific Psychology · doi:10.1037/arc0000007 · Cited by 18
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161
“Maxwell, Lau, and Howard (2015) suggest that replication studies demonstrate the absence of an effect by using equivalence bounds of D L ¼ (cid:2) .1 and D U ¼ .1 or even D L ¼ (cid:2) .05 and D U ¼ .05.”
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
Lack of group-to-individual generalizability is a threat to human subjects research
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711978115 · Cited by 1,099
Metadata
- Authors
- Maxwell, S. E., Lau, M. Y., & Howard, G. S.
- Venue
- American Psychologist
- Published
- 2015
- Volume
- 70
- Issue
- 6
- Pages
- 487-498
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W1470195681
- Cited by
- 1,005
- References
- 65
- FWCI
- 62.49
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.