Replications in Psychology Research: How Often Do They Really Occur?
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Recent controversies in psychology have spurred conversations about the nature and quality of psychological research. One topic receiving substantial attention is the role of replication in psychological science. Using the complete publication history of the 100 psychology journals with the highest 5-year impact factors, the current article provides an overview of replications in psychological research since 1900. This investigation revealed that roughly 1.6% of all psychology publications used the term replication in text. A more thorough analysis of 500 randomly selected articles revealed that only 68% of articles using the term replication were actual replications, resulting in an overall replication rate of 1.07%. Contrary to previous findings in other fields, this study found that the majority of replications in psychology journals reported similar findings to their original studies (i.e., they were successful replications). However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. Moreover, despite numerous systemic biases, the rate at which replications are being published has increased in recent decades.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [13]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [13]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [13]
In the citing paper
“For example, it has been estimated that less than 1% of all psychological research is ever replicated [49]”
From The natural selection of bad science (2016) · cites this paper as [49]
In the citing paper
“This is consistent with empirical estimations of replication rates in psychology [49].”
From The natural selection of bad science (2016) · cites this paper as [49]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
(1963) · Medical Entomology and Zoology · Cited by 15,085
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Contrary to previous findings in other fields (e.g., Ioannidis, 2005), this study found that the vast majority of both direct and conceptual replications in psychology journals reported similar findings to their original studies (i.e., successful replications).”
ResultStatistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 777
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-027 · Cited by 652
The Design of Replicated Studies
Lindsay, R. M., & Ehrenberg, A. S. C. (1993) · The American Statistician · doi:10.1080/00031305.1993.10475983 · Cited by 490
Allport, G. W., & Kramer, B. M. (1946) · The Journal of Psychology · doi:10.1080/00223980.1946.9917293 · Cited by 426
Replications and extensions in marketing: Rarely published but quite contrary
Hubbard, R., & Armstrong, J. S. (1994) · International Journal of Research in Marketing · doi:10.1016/0167-8116(94)90003-5 · Cited by 416
Analyzing data: Sanctification or detective work?
Tukey, J. W. (1969) · American Psychologist · doi:10.1037/h0027108 · Cited by 313
Replication research's disturbing trend
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2006.12.003 · Cited by 267
Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011) · Science · doi:10.1126/science.334.6060.1225 · Cited by 245
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 123
The Design of Replicated Studies
Lindsay, R. M., & Ehrenberg, A. S. C. (1993) · The American Statistician · doi:10.2307/2684982 · Cited by 118
Reviewer bias against replication research.
Neuliep, J. W., & Crandall, R. (1993) · Journal of social behavior and personality · Cited by 113
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 102
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037//0003-066x.44.10.1276 · Cited by 91
Crocker, J., & Cooper, M. L. (2011) · Science · doi:10.1126/science.1216775 · Cited by 88
REPLICATION IN EXPERIMENTAL COMMUNICATION RESEARCH: AN ANALYSIS
Kelly, C. W., Chase, L. J., & Tucker, R. K. (1979) · Human Communication Research · doi:10.1111/j.1468-2958.1979.tb00646.x · Cited by 59
Do We Really Need More Replications?
Hunt, K. P. (1975) · Psychological Reports · doi:10.2466/pr0.1975.36.2.587 · Cited by 53
A replication of "some roots of prejudice."
Rosenblith, J. F. (1949) · Journal of Abnormal & Social Psychology · doi:10.1037/h0055399 · Cited by 50
Research works printed but not linked: 2
- Collins H. M., 1985, Changing order
- 10.1037/h0027 · doi:10.1037/h0027
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Racial Inequality in Psychological Research: Trends of the Past and Recommendations for the Future
Roberts, S. O., Bareket-Shavit, C., Dollins, F. A., Goldie, P. D., & Mortenson, E. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620927709 · Cited by 1,049
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Metadata
- Authors
- Makel, M. C., Plucker, J. A., & Hegarty, B.
- Affiliations
- Duke University (US)
- University of Connecticut (US)
- University of New Hampshire (US)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-11
- Volume
- 7
- Issue
- 6
- Pages
- 537-542
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W1987673761
- Cited by
- 750
- References
- 28
- Citations by year
- 2026: 21 · 2025: 35 · 2024: 38 · 2023: 48 · 2022: 46 · 2021: 102 · 2020: 63 · 2019: 77 · 2018: 77 · 2017: 54 · 2016: 57 · 2015: 51 · 2014: 44 · 2013: 32 · 2012: 3
- FWCI
- 55.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- replication, replication rate, psychology journals, publishing history, publication bias, research quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.