CoChat
Sign Up
PreprintCited by 52Death Anxiety and Social ExclusionMeta-analysis and systematic reviewsPsychology Research and Bibliometrics

A Word of Caution about Many Labs 4: If You Fail to Follow Your Preregistered Plan, You May Fail to Find a Real Effect

Chatard, Armand, Hirschberger, Gilad, Pyszczynski, Tom·2020
DOI 10.31234/osf.io/ejubn
Open this paper in CoChat — free account

Abstract

A team of 37 researchers (Many Labs 4; Klein et al., 2019) has recently reported that it has failed to replicate the effect of mortality salience on worldview defense – a classic finding from terror management theory (TMT). This collaborative project (21 labs, N = 2220) has the potential to provide useful information regarding the robustness of an often-replicated and influential empirical finding. However, Klein et al. (2019) deviated from their preregistered plan by including smaller samples than specified in their pre-registration. This undisclosed deviation from their plan is problematic because it undermines the potential benefits of pre-registration; indeed, the negative results of their meta-analyses appear to be predominantly driven by small studies. We reanalyzed the Many Labs 4 data after excluding studies that did not meet the preregistered minimum sample size (40 participants per cell). Results showed that the data actually do replicate the original study. This successful replication emerged only in the expert advice variation of these studies, which addresses one of the purposes of this study by showing that replication is more likely to occur when researchers follow the advice of researchers with considerable experience in this domain. We discuss the importance of following preregistered plans to avoid misleading conclusions and potential issues involved with literal replications of effects that may depend on societal zeitgeist at the time data are collected.

Integrity signals

Registry

Crossref · 10.31234/osf.io/ejubn

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Group-based changes in what is considered the appropriate in-group norm may explain nonreplications of MS effects.

    “At the same time, group-based changes in what is considered the appropriate in-group norm may explain nonreplications of MS effects (e.g., on derogating critics of the United States; Chatard et al., 2020).”

    From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Chatard et al., 2020

  • Chatard, Hirschberger, and Pyszczynski (2020) re-analysis found that the key effect weakly replicated (Hedges' g = 0.13, p = .045) in studies that followed the advice of the original authors, at least when one particular set of exclusion criteria was applied.

    “When these authors re-analyzed the data of only those studies that met the minimum preregistered sample size they found that the key effect weakly replicated in the studies that followed the advice of the original authors (Hedges' g = 0.13, p = .045), at least when one particular set of exclusion criteria was applied.”

    From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Chatard, Hirschberger, and Pyszczynski (2020)

  • A re-analysis (Chatard et al., 2020) focusing only on studies that met preregistered criteria found significant evidence for replication when the studies followed the guidance of the original authors and one set of exclusion criteria were applied.

    “however, a re-analysis (Chatard et al., 2020) that focused only on the studies that met preregistered criteria found significant evidence for replication when the studies followed the guidance of the original authors and one set of exclusion criteria were applied.”

    From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Chatard et al., 2020

  • Other sets of exclusion criteria in the Chatard et al. re-analysis did not reveal evidence of replication.

    “In contrast, other sets of exclusion criteria in this re-analysis did not reveal evidence of replication.”

    From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Chatard et al., 2020

  • Those author-advised location effects were significantly different from zero with one particular set of exclusion criteria for studies that met all preregistered criteria (Chatard et al., 2020).

    “(and were significantly different from zero with one particular set of exclusion criteria for those studies that met all preregistered criteria; Chatard et al., 2020)”

    From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Chatard et al., 2020

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Chatard, A., Hirschberger, G., & Pyszczynski, T.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • Reichman University (IL)
Corresponding author
Chatard, Armand
ORCID
Publisher
Center for Open Science
Published
2020-02-07
DOI
10.31234/osf.io/ejubn
OpenAlex
W3163230172
Cited by
51
References
21
Citations by year
2026: 4 · 2025: 4 · 2024: 4 · 2023: 14 · 2022: 10 · 2021: 8 · 2020: 7
Type
posted content
Language
en
Open access
closed
Locations
PsyArXiv (OSF Preprints) (submittedVersion)
Topics
  • Social Sciences › Psychology › Social Psychology › Death Anxiety and Social Exclusion
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
preregistration, replication, mortality salience, worldview defense, terror management theory, sample size, small sample bias, expert advice, reanalysis, replication robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review