The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Investigating Variation in Replicability: A “Many Labs” Replication Project
Open accessPublished version of a preprint
Abstract
Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using similar criteria, the”
From Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement (2022) · cites this paper as Klein et al., 2014
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A ' Many labs ' project reported that 10 of 13 studies taken from the psychological literature had been consistently replicated multiple times across different settings (21)”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“Nonetheless, none of these efforts guarantee that unanticipated factors in the sample, setting, or procedure altered the observed effect magnitudes ( 32 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [32]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, this perspective is not universally shared [8], and there are examples of EBI scale-ups where health outcomes did not improve as intended [9].”
From “Scaling-out” evidence-based interventions to new populations or new health care delivery systems (2017) · cites this paper as [9]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 2×
“Commentaries and a rejoinder on this paper are available (Crisp, Miles, & Husnu, 2014; Ferguson, Carter, & Hassin, 2014; Kahneman, 2014; Klein et al., 2014; Monin & Oppenheimer, 2014; Schwarz & Strack, 2014; doi: 10.1027/1864-9335/a000202).”
InfluentialWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Mental Accounting and Consumer Choice
Thaler, R. (1985) · Marketing Science · doi:10.1287/mksc.4.3.199 · Cited by 4,885
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Direct replication is infrequent, resulting in a published literature that sustains spurious findings (Ioannidis, 2005) and a lack of identification of the eliciting conditions for an effect.”
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citationInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441
Mental Accounting and Consumer Choice
Thaler, R. H. (2008) · Marketing Science · doi:10.1287/mksc.1070.0330 · Cited by 3,415
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 1×
“…interesting considering that there is substantial interest and commentary about the contingency of effects on our two moderators, lab versus online (Gosling, Vazire, Srivastava, & John, 2004; Paolacci, Chandler, & Ipeirotis, 2010), and cultural variation across nations (Henrich et al., 2010).”
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125
Research works printed but not linked: 2
- Hyman H. H., 1950, The teaching of contemporary affairs: 21st yearbook of the National Council of Social Studies, 11
- Klein, R. A. Ratliff, K. A. Vianello, M. Adams, R. B. Jr. Bahník, Š. Bernstein, M. J. … Nosek, B. A. (2014). Theory building through replication: Response to commentaries on the “Many Labs” replication project. Commentaries and rejoinder on Klein et al. (2014). Social Psychology . Advance online publication. doi: 10.1027/1864-9335/1000202 · doi:10.1027/1864-9335/1000202
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationBeyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citationChoosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
Metadata
- Authors
- Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Huntsinger, J. R., IJzerman, H., John, M.-S., Joy-Gaba, J. A., Barry Kappes, H., Krueger, L. E., Kurtz, J., Levitan, C. A., Mallett, R. K., Morris, W. L., Nelson, A. J., Nier, J. A., Packard, G., Pilati, R., Rutchick, A. M., Schmidt, K., Skorinko, J. L., Smith, R., Steiner, T. G., Storbeck, J., Van Swol, L. M., Thompson, D., van ‘t Veer, A. E., Ann Vaughn, L., Vranka, M., Wichman, A. L., Woodzicka, J. A., & Nosek, B. A.
- Affiliations
- University of Florida (US)
- University of Padua (IT)
- Pennsylvania State University (US)
- University of Würzburg (DE)
- Uniwersytet SWPS (PL)
- Tilburg University (NL)
- City University of New York (US)
- Koç University, Istanbul, Turkey
- University of Michigan (US)
- HELP University (MY)
- Texas A&M University (US)
- San Diego State University (US)
- Mount Saint Vincent University (CA)
- Radboud University Nijmegen (NL)
- Virginia Commonwealth University (US)
- East Texas A&M University (US)
- Loyola University Chicago (US)
- Worcester Polytechnic Institute (US)
- London School of Economics and Political Science, London, UK
- James Madison University (US)
- Occidental College (US)
- McDaniel College (US)
- Connecticut College (US)
- Wilfrid Laurier University (CA)
- Universidade de Brasília (BR)
- California State University, Northridge (US)
- University of Virginia (US)
- The Ohio State University (US)
- University of Wisconsin-Madison, WI, USA
- Tilburg University, The Netherlands
- Ithaca College, NY, USA
- Charles University (CZ)
- Western Kentucky University (US)
- Washington and Lee University (US)
- Center for Open Science (US)
- ORCID
- Klein, Richard A.
- Ratliff, Kate A.
- Vianello, Michelangelo
- Adams, Reginald B.
- Bahník, Štěpán
- Bernstein, Michael J.
- Bocian, Konrad
- Brandt, Mark J.
- Brumbaugh, Claudia Chloe
- Chandler, Jesse
- Devos, Thierry
- Frankowska, Natalia
- Galliani, Elisa Maria
- Hasselman, Fred
- Hicks, Joshua A.
- IJzerman, Hans
- John, Melissa-Sue
- Joy-Gaba, Jennifer A.
- Krueger, Lacy E.
- Levitan, Carmel A.
- Mallett, Robyn K.
- Morris, Wendy L.
- Nelson, Anthony J.
- Packard, Grant
- Pilati, Ronaldo
- Rutchick, Abraham M.
- Schmidt, Kathleen
- Skorinko, Jeanine L.
- Smith, Robert
- Steiner, Troy G.
- Vranka, Marek
- Wichman, Aaron L.
- Woodzicka, Julie A.
- Nosek, Brian A.
- Venue
- Social Psychology
- Publisher
- Hogrefe Publishing Group
- Published
- 2014-05-01
- Volume
- 45
- Issue
- 3
- Pages
- 142-152
- ISSN
- 1864-9335 (ISSN-L)
- 2151-2590
- OpenAlex
- W1941783251
- Cited by
- 1,208
- References
- 41
- Citations by year
- 2026: 62 · 2025: 62 · 2024: 73 · 2023: 97 · 2022: 103 · 2021: 143 · 2020: 151 · 2019: 135 · 2018: 114 · 2017: 86 · 2016: 88 · 2015: 62 · 2014: 28
- FWCI
- 286.88
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Social Psychology (publishedVersion, cc-by-nc-nd)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (acceptedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Tilburg University Research Portal (submittedVersion, other-oa)
- Research portal (Tilburg University) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- CSUN ScholarWorks (California State University, Northridge) (submittedVersion)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Topics
- Social Sciences › Sociology and Political Science › Social and Intergroup Psychology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- replicability, replication, direct replication, imagined contact, prejudice, conservatism, system justification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating Variation in Replicability: A “Many Labs” Replication Project. https://app.cochat.ai/papers/10.1027/1864-9335/a000178
- The paper
- Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., . . . Nosek, B. A. (2014). Investigating Variation in Replicability: A “Many Labs” Replication Project. Social Psychology. https://doi.org/10.1027/1864-9335/a000178
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.