CoChat
Sign Up
Cited by 944Meta-analysis and systematic reviewsPsychology Research and BibliometricsReliability and Agreement in Measurement

The DOI matches its Crossref record; 146 of the 146 references it cites were found in a registry or an index.

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Open access
Klein, Richard A., Vianello, Michelangelo, Hasselman, Fred, Adams, Byron G., Adams, Reginald B., Alper, Sinan, Aveyard, Mark, Axt, Jordan R., Babalola, Mayowa T., Bahník, Štěpán, Batra, Rishtee, Berkics, Mihály, Bernstein, Michael J., Berry, Daniel R., Bialobrzeska, Olga, Binan, Evans Dami, Bocian, Konrad, Brandt, Mark J., Busching, Robert, Rédei, Anna Cabak, Cai, Huajian, Cambier, Fanny, Cantarero, Katarzyna, Carmichael, Cheryl L., Ceric, Francisco, Chandler, Jesse, Chang, Jen-Ho, Chatard, Armand, Chen, Eva E., Cheong, Winnee, Cicero, David C., Coen, Sharon, Coleman, Jennifer A., Collisson, Brian, Conway, Morgan A., Corker, Katherine S., Curran, Paul G., Cushman, Fiery, Dagona, Zubairu K., Dalgar, Ilker, Dalla Rosa, Anna, Davis, William E., de Bruijn, Maaike, De Schutter, Leander, Devos, Thierry, de Vries, Marieke, Doğulu, Canay, Dozo, Nerisa, Dukes, Kristin Nicole, Dunham, Yarrow, Durrheim, Kevin, Ebersole, Charles R., Edlund, John E., Eller, Anja, English, Alexander Scott, Finck, Carolyn, Frankowska, Natalia, Freyre, Miguel-Ángel, Friedman, Mike, Galliani, Elisa Maria, Gandi, Joshua C., Ghoshal, Tanuka, Giessner, Steffen R., Gill, Tripat, Gnambs, Timo, Gómez, Ángel, González, Roberto, Graham, Jesse, Grahe, Jon E., Grahek, Ivan, Green, Eva G. T., Hai, Kakul, Haigh, Matthew, Haines, Elizabeth L., Hall, Michael P., Heffernan, Marie E., Hicks, Joshua A., Houdek, Petr, Huntsinger, Jeffrey R., Huynh, Ho Phi, IJzerman, Hans, Inbar, Yoel, Innes-Ker, Åse H., Jiménez-Leal, William, John, Melissa-Sue, Joy-Gaba, Jennifer A., Kamiloğlu, Roza G., Kappes, Heather Barry, Karabati, Serdar, Karick, Haruna, Keller, Victor N., Kende, Anna, Kervyn, Nicolas, Knežević, Goran, Kovacs, Carrie, Krueger, Lacy E., Kurapov, German, Kurtz, Jamie, Lakens, Daniël, Lazarević, Ljiljana B., Levitan, Carmel A., Lewis, Neil A., Lins, Samuel, Lipsey, Nikolette P., Losee, Joy E., Maassen, Esther, Maitner, Angela T., Malingumu, Winfrida, Mallett, Robyn K., Marotta, Satia A., Međedović, Janko, Mena-Pacheco, Fernando, Milfont, Taciano L., Morris, Wendy L., Murphy, Sean C., Myachykov, Andriy, Neave, Nick, Neijenhuijs, Koen, Nelson, Anthony J., Neto, Félix, Lee Nichols, Austin, Ocampo, Aaron, O’Donnell, Susan L., Oikawa, Haruka, Oikawa, Masanori, Ong, Elsie, Orosz, Gábor, Osowiecka, Malgorzata, Packard, Grant, Pérez-Sánchez, Rolando, Petrović, Boban, Pilati, Ronaldo, Pinter, Brad, Podesta, Lysandra, Pogge, Gabrielle, Pollmann, Monique M. H., Rutchick, Abraham M., Saavedra, Patricio, Saeri, Alexander K., Salomon, Erika, Schmidt, Kathleen, Schönbrodt, Felix D., Sekerdej, Maciej B., Sirlopú, David, Skorinko, Jeanine L. M., Smith, Michael A., Smith-Castro, Vanessa, Smolders, Karin C. H. J., Sobkow, Agata, Sowden, Walter, Spachtholz, Philipp, Srivastava, Manini, Steiner, Troy G., Stouten, Jeroen, Street, Chris N. H., Sundfelt, Oskar K., Szeto, Stephanie, Szumowska, Ewa, Tang, Andrew C. W., Tanzer, Norbert, Tear, Morgan J., Theriault, Jordan, Thomae, Manuela, Torres, David, Traczyk, Jakub, Tybur, Joshua M., Ujhelyi, Adrienn, van Aert, Robbie C. M., van Assen, Marcel A. L. M., van der Hulst, Marije, van Lange, Paul A. M., van ’t Veer, Anna Elisabeth, Vásquez- Echeverría, Alejandro, Ann Vaughn, Leigh, Vázquez, Alexandra, Vega, Luis Diego, Verniers, Catherine, Verschoor, Mark, Voermans, Ingrid P. J., Vranka, Marek A., Welch, Cheryl, Wichman, Aaron L., Williams, Lisa A., Wood, Michael, Woodzicka, Julie A., Wronska, Marta K., Young, Liane, Zelenski, John M., Zhijia, Zeng, Nosek, Brian A.·2018·Advances in Methods and Practices in Psychological Science
DOI 10.1177/2515245918810225
Open this paper in CoChat — free accountOpen PDF

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/2515245918810225

Cites

146 of 146 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 142 references

Research works printed but not linked: 4

  1. 10.1027/1864-9335/a000202 · DOI
  2. Klein R. A., 2014, Social Psychology, 45, 307, 10.1027/1864-9335/a000178 · DOI · Paper page
  3. 10.1111/1467-8284.00419 · DOI · Paper page
  4. 10.1027/1864-9335/a000202 · DOI

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., Rédei, A. C., Cai, H., Cambier, F., Cantarero, K., Carmichael, C. L., Ceric, F., Chandler, J., Chang, J.-H., Chatard, A., Chen, E. E., Cheong, W., Cicero, D. C., Coen, S., Coleman, J. A., Collisson, B., Conway, M. A., Corker, K. S., Curran, P. G., Cushman, F., Dagona, Z. K., Dalgar, I., Dalla Rosa, A., Davis, W. E., de Bruijn, M., De Schutter, L., Devos, T., de Vries, M., Doğulu, C., Dozo, N., Dukes, K. N., Dunham, Y., Durrheim, K., Ebersole, C. R., Edlund, J. E., Eller, A., English, A. S., Finck, C., Frankowska, N., Freyre, M.-Á., Friedman, M., Galliani, E. M., Gandi, J. C., Ghoshal, T., Giessner, S. R., Gill, T., Gnambs, T., Gómez, Á., González, R., Graham, J., Grahe, J. E., Grahek, I., Green, E. G. T., Hai, K., Haigh, M., Haines, E. L., Hall, M. P., Heffernan, M. E., Hicks, J. A., Houdek, P., Huntsinger, J. R., Huynh, H. P., IJzerman, H., Inbar, Y., Innes-Ker, Å. H., Jiménez-Leal, W., John, M.-S., Joy-Gaba, J. A., Kamiloğlu, R. G., Kappes, H. B., Karabati, S., Karick, H., Keller, V. N., Kende, A., Kervyn, N., Knežević, G., Kovacs, C., Krueger, L. E., Kurapov, G., Kurtz, J., Lakens, D., Lazarević, L. B., Levitan, C. A., Lewis, N. A., Lins, S., Lipsey, N. P., Losee, J. E., Maassen, E., Maitner, A. T., Malingumu, W., Mallett, R. K., Marotta, S. A., Međedović, J., Mena-Pacheco, F., Milfont, T. L., Morris, W. L., Murphy, S. C., Myachykov, A., Neave, N., Neijenhuijs, K., Nelson, A. J., Neto, F., Lee Nichols, A., Ocampo, A., O’Donnell, S. L., Oikawa, H., Oikawa, M., Ong, E., Orosz, G., Osowiecka, M., Packard, G., Pérez-Sánchez, R., Petrović, B., Pilati, R., Pinter, B., Podesta, L., Pogge, G., Pollmann, M. M. H., Rutchick, A. M., Saavedra, P., Saeri, A. K., Salomon, E., Schmidt, K., Schönbrodt, F. D., Sekerdej, M. B., Sirlopú, D., Skorinko, J. L. M., Smith, M. A., Smith-Castro, V., Smolders, K. C. H. J., Sobkow, A., Sowden, W., Spachtholz, P., Srivastava, M., Steiner, T. G., Stouten, J., Street, C. N. H., Sundfelt, O. K., Szeto, S., Szumowska, E., Tang, A. C. W., Tanzer, N., Tear, M. J., Theriault, J., Thomae, M., Torres, D., Traczyk, J., Tybur, J. M., Ujhelyi, A., van Aert, R. C. M., van Assen, M. A. L. M., van der Hulst, M., van Lange, P. A. M., van ’t Veer, A. E., Vásquez- Echeverría, A., Ann Vaughn, L., Vázquez, A., Vega, L. D., Verniers, C., Verschoor, M., Voermans, I. P. J., Vranka, M. A., Welch, C., Wichman, A. L., Williams, L. A., Wood, M., Woodzicka, J. A., Wronska, M. K., Young, L., Zelenski, J. M., Zhijia, Z., & Nosek, B. A.
Affiliations
  • Laboratoire Inter-universitaire de Psychologie: Personnalité, Cognition, Changement Social (FR)
  • University of Padua (IT)
  • Radboud University Nijmegen (NL)
  • Tilburg University (NL)
  • Pennsylvania State University (US)
  • Yaşar University (TR)
  • American University of Sharjah (AE)
  • Duke University (US)
  • United Arab Emirates University (AE)
  • Prague University of Economics and Business (CZ)
  • Saint Joseph's University (US)
  • Eötvös Loránd University (HU)
  • California State University, San Marcos (US)
  • Warsaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
  • Department of General and Applied Psychology, University of Jos
  • Uniwersytet SWPS (PL)
  • University of Potsdam (DE)
  • Lund University (SE)
  • Institute of Psychology, Chinese Academy of Sciences (CN)
  • UCLouvain (BE)
  • The Graduate Center, CUNY (US)
  • Facultad de Psicologia, Universidad del Desarrollo
  • University of Michigan (US)
  • National Taiwan University (TW)
  • Centre National de la Recherche Scientifique (FR)
  • Hong Kong University of Science and Technology (HK)
  • HELP University (MY)
  • University of Hawaiʻi at Mānoa (US)
  • University of Salford (GB)
  • Virginia Commonwealth University (US)
  • Azusa Pacific University (US)
  • University of Florida (US)
  • Grand Valley State University (US)
  • Harvard University (US)
  • University of Jos (NG)
  • Department of Psychology, Middle East Technical University
  • Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua
  • Wittenberg University (US)
  • Department of Social Psychology, Tilburg University
  • Leadership and Human Resource Management, WHU – Otto Beisheim School of Management
  • San Diego State University (US)
  • Behavioural Science Institute, Radboud University Nijmegen
  • Başkent University (TR)
  • The University of Queensland (AU)
  • Allegheny College (US)
  • Yale University (US)
  • University of KwaZulu-Natal (ZA)
  • University of Virginia (US)
  • Rochester Institute of Technology (US)
  • Universidad Nacional Autónoma de México (MX)
  • Shanghai International Studies University (CN)
  • Universidad de Los Andes (CO)
  • Baruch College (US)
  • Erasmus University Rotterdam (NL)
  • Wilfrid Laurier University (CA)
  • Johannes Kepler University of Linz (AT)
  • Universidad Nacional de Educación a Distancia (ES)
  • Pontificia Universidad Católica de Chile (CL)
  • University of Utah (US)
  • Pacific Lutheran University (US)
  • Ghent University (BE)
  • University of Lausanne (CH)
  • Amity University (IN)
  • Northumbria University (GB)
  • William Paterson University (US)
  • Lurie Children's Hospital (US)
  • Texas A&M University (US)
  • Jan Evangelista Purkyně University in Ústí nad Labem (CZ)
  • Loyola University Chicago (US)
  • Texas A&M University – San Antonio (US)
  • University of Toronto (CA)
  • Departamento de Psicología, Universidad de los Andes, Colombia
  • Worcester Polytechnic Institute (US)
  • Department of Psychology, University of Amsterdam
  • London School of Economics and Political Science (GB)
  • Department of Business Administration, Istanbul Bilgi University
  • Universidade de Brasília (BR)
  • University of Belgrade (RS)
  • East Texas A&M University (US)
  • James Madison University (US)
  • Eindhoven University of Technology (NL)
  • Occidental College (US)
  • Cornell University (US)
  • Universidade do Porto (PT)
  • Open University of Tanzania (TZ)
  • Tufts University (US)
  • Singidunum University (RS)
  • Latin University of Costa Rica (CR)
  • Victoria University of Wellington (NZ)
  • McDaniel College (US)
  • The University of Melbourne (AU)
  • Amsterdam Public Health (NL)
  • Department of Psychology, University of Central Florida
  • George Fox University (US)
  • Doshisha University (JP)
  • Hong Kong Metropolitan University (HK)
  • Stanford University (US)
  • Institute for Psychological Research, University of Costa Rica
  • Institute of Criminological & Sociological Research (RS)
  • California State University, Northridge (US)
  • University of Sussex (GB)
  • Monash University (AU)
  • University of Chicago (US)
  • Southern Illinois University Carbondale (US)
  • Ludwig-Maximilians-Universität München (DE)
  • Jagiellonian University (PL)
  • Universidad del Desarrollo (CL)
  • Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities
  • Walter Reed Army Institute of Research (US)
  • University of Regensburg (DE)
  • KU Leuven (BE)
  • University of Huddersfield (GB)
  • University of Graz (AT)
  • Northeastern University (US)
  • University of Winchester (GB)
  • Universidad Iberoamericana (MX)
  • Vrije Universiteit Amsterdam (NL)
  • Department of Methodology and Statistics, Tilburg University
  • Department of Obstetrics and Gynaecology, Erasmus MC, Rotterdam, The Netherlands
  • Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam
  • Methodology and Statistics Unit, Institute of Psychology, Leiden University
  • Centro de Investigación Básica en Psicología, Universidad de la República
  • Department of Psychology, Ithaca College
  • Sorbonne Paris Cité (FR)
  • University of Groningen (NL)
  • Charles University (CZ)
  • Western Kentucky University (US)
  • UNSW Sydney (AU)
  • Washington and Lee University (US)
  • Boston College (US)
  • Carleton University (CA)
  • Zhejiang University of Finance and Economics (CN)
  • Center for Open Science (US)
Corresponding author
Klein, Richard A.
ORCID
Venue
Advances in Methods and Practices in Psychological Science
Publisher
SAGE Publications
Published
2018-12
Volume
1
Issue
4
Pages
443-490
ISSN
  • 2515-2459 (ISSN-L)
  • 2515-2467
DOI
10.1177/2515245918810225
OpenAlex
W2776961836
Cited by
944
References
142
Citations by year
2026: 44 · 2025: 76 · 2024: 119 · 2023: 147 · 2022: 135 · 2021: 152 · 2020: 155 · 2019: 103 · 2018: 9
FWCI
79.88
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Advances in Methods and Practices in Psychological Science (publishedVersion)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (acceptedVersion)
  • Northumbria Research Link (Northumbria University) (acceptedVersion)
  • TU/e Research Portal (acceptedVersion, other-oa)
  • London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
  • Journal of International Crisis and Risk Communication Research (acceptedVersion)
  • TU/e Research Portal (acceptedVersion)
  • UvA-DARE (University of Amsterdam) (acceptedVersion, other-oa)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (acceptedVersion, other-oa)
  • Leiden Repository (Leiden University) (acceptedVersion)
  • Huddersfield Research Portal (University of Huddersfield) (acceptedVersion, other-oa)
  • VU Research Portal (acceptedVersion, other-oa)
  • Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) (acceptedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (acceptedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Leiden Repository (Leiden University) (submittedVersion)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion, other-oa)
  • Lirias (submittedVersion, other-oa)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Digital Commons - George Fox University (George Fox University) (submittedVersion)
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • Homo Politicus (Academy of Humanities and Economics in Lodz) (submittedVersion, cc-by-nc-nd)
  • Tilburg University Research Portal (submittedVersion, other-oa)
  • Research portal (Tilburg University) (submittedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • Americanae (AECID Library) (submittedVersion, other-oa)
  • Digital Access to Libraries (submittedVersion, other-oa)
  • Lund University Publications (Lund University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • VU Research Portal (submittedVersion, other-oa)
  • UNSWorks (University of New South Wales, Sydney, Australia) (submittedVersion, cc-by-nc-nd)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • TU/e Research Portal (submittedVersion, other-oa)
Funders
Association for Psychological Science
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
replication, effect size heterogeneity, setting effects, pre-registered replication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review