Large-Scale Replication Projects in Contemporary Psychological Research
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Replication is complicated in psychological research because studies of a given psychological phenomenon can never be direct or exact replications of one another, and thus effect sizes vary from one study of the phenomenon to the next—an issue of clear importance for replication. Current large-scale replication projects represent an important step forward for assessing replicability, but provide only limited information because they have thus far been designed in a manner such that heterogeneity either cannot be assessed or is intended to be eliminated. Consequently, the nontrivial degree of heterogeneity found in these projects represents a lower bound on the true degree of heterogeneity. We recommend enriching large-scale replication projects going forward by embracing heterogeneity. We argue this is the key for assessing replicability: if effect sizes are sufficiently heterogeneous—even if the sign of the effect is consistent—the phenomenon in question does not seem particularly replicable and the theory underlying it seems poorly constructed and in need of enrichment. Uncovering why and revising theory in light of it will lead to improved theory that explains heterogeneity and increases replicability. Given this, large-scale replication projects can play an important role not only in assessing replicability but also in advancing theory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“…that a Constraints on Generality statement provides the necessary information to allow for inference, via hierarchical modeling and poststratification (Park, Gelman, and Bafumi 2004), regarding how quantities of interest might vary along with the key method factors identified in the statement.”
BackgroundAssessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Bargh, J. A., Chen, M., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.71.2.230 · Cited by 2,736
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986 · cited 2×
“More systematically, Simons, Shoda, and Lindsay (2017) make the excellent suggestion that all primary research papers include a “Constraints on Generality” statement that identifies key method factors, thereby “help[ing] other researchers...when conducting a direct replication” and “encourag[ing]…”
BackgroundBem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · cited 1×
“This problem appears particularly acute in psychology, where the failure to replicate several prominent findings (for example, Bargh, Chen, and Burrows (1996); Bargh et al. (2001); Carney, Cuddy, and Yap (2010); Bem (2011)) has attracted the attention of both academics and the popular press.”
BackgroundChoice overload: A conceptual review and meta-analysis
Chernev, A., Böckenholt, U., & Goodman, J. (2015) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2014.08.002 · Cited by 858
Self-citationThe alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls
Park, D. K., Gelman, A., & Bafumi, J. (2004) · Political Analysis · doi:10.1093/pan/mph024 · Cited by 480 · cited 1×
“To place I2 in context, Pigott (2012) defines low, medium, and high heterogeneity in psychological research as I2 of 25%, 50%, and 75% respectively (see also Higgins and Thompson 2002).”
BackgroundSelf-citation
Cited by
Moving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
What can be learned from growth mindset controversies?
Yeager, D. S., & Dweck, C. S. (2020) · American Psychologist · doi:10.1037/amp0000794 · Cited by 868
“Therefore the Rienzo study is not exactly evidence against mindset effects (see a discussion of this statistical argument in McShane et al., 2019).”
BackgroundBehavioural science is unlikely to change the world without a heterogeneity revolution
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01143-3 · Cited by 537
Using and Interpreting Fixed Effects Models
BREUER, M., & DEHAAN, E. (2024) · Journal of Accounting Research · doi:10.1111/1475-679x.12559 · Cited by 526
Soto, C. J. (2019) · Psychological Science · doi:10.1177/0956797619831612 · Cited by 470
Metadata
- Authors
- McShane, B. B., Tackett, J. L., Böckenholt, U., & Gelman, A.
- Venue
- The American Statistician
- Publisher
- Informa UK Limited
- Published
- 2019-03-20
- Volume
- 73
- Issue
- sup1
- Pages
- 99-105
- ISSN
- 0003-1305 (ISSN-L)
- 1537-2731
- OpenAlex
- W2766894915
- Cited by
- 104
- References
- 16
- FWCI
- 9.51
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large-Scale Replication Projects in Contemporary Psychological Research. https://app.cochat.ai/papers/10.1080/00031305.2018.1505655
- The paper
- McShane, B. B., Tackett, J. L., Böckenholt, U., & Gelman, A. (2019). Large-Scale Replication Projects in Contemporary Psychological Research. The American Statistician. https://doi.org/10.1080/00031305.2018.1505655
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.