Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Over the last few years, psychology researchers have become increasingly preoccupied with the question of whether findings from psychological studies are generally replicable. The debates have originated from some unfortunate events of scientific misconduct in the field, and they have reached a climax with the recent discovery of a relatively weak rate of replicability of published literature, leading to the so-called replicability crisis in psychology. The present paper is concerned with examining the issue of replicability in the field of social psychology. We begin by drawing a state of the art of the crisis in this field. We then highlight some possible causes for the crisis, discussing topics of statistical power, questionable research practices, publication standards, and hidden auxiliary assumptions of context-dependency of social psychological theories. Finally, we argue that given the absence of absolute falsification in science, social psychology could greatly benefit from adopting McGuire’s perspectivist approach to knowledge construction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 2×
“Crucially, however, one should always keep in mind that no single statistical index can ever substitute for well-informed scientific thinking (Gigerenzer & Marewski, 2015; Wasserstein & Lazar, 2016).”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“Undoubtedly, this heightened interest follows the recent debates that called into question conventional practices of research conduct and data analysis (e.g., Ioannidis, 2005).”
BackgroundCulture and systems of thought: Holistic versus analytic cognition.
Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001) · Psychological Review · doi:10.1037/0033-295x.108.2.291 · Cited by 4,263
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
The Methodology of Scientific Research Programmes: Philosophical Papers
Lakatos, I. (1978) · doi:10.1017/cbo9780511621123 · Cited by 3,045
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · cited 1×
“The reality is that the better a journal’s reputation – based on indices such as the journal impact factor (Garfield, 2006) – the harsher the competition to publish in this journal becomes.”
BackgroundBargh, J. A., Chen, M., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.71.2.230 · Cited by 2,736
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Gergen, K. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034436 · Cited by 1,845 · cited 1×
“In some way, current debates on replicability issues related to context-dependency echo the intellectual crisis from the 1970s that called into question the epistemological foundations of the mainstream experimental social psychology (e.g., Gergen, 1973).”
BackgroundThe statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
The SAGE Handbook of Quantitative Methodology for the Social Sciences
Kaplan, D. (2004) · doi:10.4135/9781412986311 · Cited by 1,454
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 2×
“This ultimately results in the disappearance of negative results from the literature (Fanelli, 2012).”
BackgroundResultShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 2×
“Until recently, replicating existing studies has been no more than just a rare phenomenon in psychological science (Schmidt, 2009).”
BackgroundTheory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938 · cited 1×
“…neither evidence for nor against the null hypothesis, and even when the latter is true, the p-value can take almost any value ( Cumming, 2012; Dienes, 2011).4 Thus, even when a theory lacks empirical support and when failed attempts at replication accumulate, it is still hardly possible to…”
BackgroundBem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · cited 2×
“Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).”
BackgroundGigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · cited 1×
“Rather, it is the way NHST is typically used in social sciences in general that would require a thorough reconsideration (see Gigerenzer, 2004; Gigerenzer, Krauss, & Vitouch, 2004).”
MethodRevised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · cited 2×
“On the one hand, recent advances in Bayesian statistics have pointed out that a p-value significant at 0.05 may often pertain to only weak evidence against the null hypothesis (Johnson, 2013; Wetzels et al., 2011).”
BackgroundInfluentialThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670
Measurement error and the replication crisis
Loken, E., & Gelman, A. (2017) · Science · doi:10.1126/science.aal3618 · Cited by 628 · cited 1×
“…assess “noisy effects” (i.e., where there are systematic random variations associated with the way variables are measured) can often yield overestimated effect sizes, whose achieved statistical significance capitalizes on random error rather than on the true parameter value (Loken & Gelman, 2017).”
BackgroundFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 3×
“…a theory lacks empirical support and when failed attempts at replication accumulate, it is still hardly possible to properly operate the falsification mechanism, and this without even mentioning the fact that publication bias may prevent failed replications to be published (Ferguson & Heene, 2012).”
BackgroundSample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · cited 1×
“Among such practices, one should remember to take into account the importance of statistical power in study design (e.g., Maxwell, 2004; Maxwell et al., 2008).”
BackgroundThe prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 1×
“For instance, Nuijten et al. (2015) reported that in a sample of more than 30,000 articles selected from the top eight psychology journals, one in eight possibly contained an inconsistent p-value that might have affected the statistical conclusion.”
BackgroundThe ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · cited 2×
“Finally, we also pointed out that the neglect of possible auxiliary assumptions relative to context-dependency variables could also account for an unexpectedly low rate of replicability in the domain of social psychology (Van Bavel et al., 2016b).”
BackgroundThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 1×
“Last but not least, methodologists have also highlighted a prevalent issue of misreporting statistical analyses in published articles (Bakker & Wicherts, 2011; Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015).”
BackgroundStatistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · cited 5×
“For instance, the mean power achieved in the Psychology Reproducibility Project (Open Science Collaboration, 2015) was estimated to be 0.92, and the researchers expected to detect around 89 effects as statistically significant based on the 100 they tested, assuming these effects were true.”
MethodInfluentialGiner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · cited 5×
“05 as a condition of publication, along with unrealistic expectations regarding researchers’ data and the need for novelty (Giner-Sorolla, 2012).”
BackgroundMethodInfluentialLakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · cited 2×
“In other words, high-powered studies hold higher informational value than low-powered studies (see Lakens & Evers, 2014, for a review).”
BackgroundCREATIVE HYPOTHESIS GENERATING IN PSYCHOLOGY: Some Useful Heuristics
McGuire, W. J. (1997) · Annual Review of Psychology · doi:10.1146/annurev.psych.48.1.1 · Cited by 203 · cited 3×
“187–188) Obviously, there is nothing wrong with conducting exploratory research per se, which should in fact occupy an integral and important part in a research program for purposes such as hypothesis generation or testing auxiliary theories (McGuire, 1997).”
BackgroundInfluentialMalignant side effects of null-hypothesis significance testing
Branch, M. (2014) · Theory & Psychology · doi:10.1177/0959354314525282 · Cited by 81 · cited 1×
“2 We hasten to make clear that this statistical tool is not bad per se, contrary to what might be thought based on the recent debates on the difficulties associated with p-values (e.g., Branch, 2014; Trafimow & Marks, 2015).”
ResultConstructing Social Psychology: Creative and Critical Aspects
McGuire, W. (1999) · doi:10.1017/cbo9780511571206 · Cited by 73
Cited by
Sommet, N., & Morselli, D. (2017) · International Review of Social Psychology · doi:10.5334/irsp.90 · Cited by 590
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 408
Über-Alienated: Powerless and Alone in the Gig Economy
Glavin, P., Bierman, A., & Schieman, S. (2021) · Work and Occupations · doi:10.1177/07308884211024711 · Cited by 157
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Leichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
Metadata
- Authors
- Świątkowski, W., & Dompnier, B.
- Venue
- International Review of Social Psychology
- Publisher
- Ubiquity Press, Ltd.
- Published
- 2017-05-02
- Volume
- 30
- Issue
- 1
- Pages
- 111
- ISSN
- 0992-986X (ISSN-L)
- 2397-8570
- OpenAlex
- W2611715956
- Cited by
- 114
- References
- 51
- FWCI
- 32.31
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.