CoChat
Sign Up
Cited by 121Misinformation and Its ImpactsCommunication and COVID-19 ImpactAcademic Publishing and Open Access

How Accurate Are Accuracy-Nudge Interventions? A Preregistered Direct Replication of Pennycook et al. (2020)

Open access
Roozenbeek, Jon, Freeman, Alexandra L. J., van der Linden, Sander·2021·Psychological Science
DOI 10.1177/09567976211024535PMC PMC8641132
Open this paper in CoChat — free account

Abstract

As part of the Systematizing Confidence in Open Research and Evidence (SCORE) program, the present study consisted of a two-stage replication test of a central finding by Pennycook et al. (2020), namely that asking people to think about the accuracy of a single headline improves “truth discernment” of intentions to share news headlines about COVID-19. The first stage of the replication test ( n = 701) was unsuccessful ( p = .67). After collecting a second round of data (additional n = 882, pooled N = 1,583), we found a small but significant interaction between treatment condition and truth discernment (uncorrected p = .017; treatment: d = 0.14, control: d = 0.10). As in the target study, perceived headline accuracy correlated with treatment impact, so that treatment-group participants were less willing to share headlines that were perceived as less accurate. We discuss potential explanations for these findings and an unreported change in the hypothesis (but not the analysis plan) from the preregistration in the original study.

AI snapshot

Written from the paper’s full text.

TL;DR

This preregistered two-stage replication of Pennycook et al. (2020) found no accuracy-nudge effect on truth discernment in Stage 1, but a small significant interaction in the pooled Stage 2 sample. The pooled effect was about half the original study’s effect, though the proposed perceived-accuracy mechanism replicated.

Key findings

  • Stage 1 (n = 701; 21,030 headline ratings) found no significant interaction between headline veracity and treatment, β = 0.0046, 95% CI [−0.016, 0.026], p = .67.
  • In the pooled sample (additional n = 882; pooled N = 1,583), the interaction was significant, β = 0.015, 95% CI [0.0027, 0.027], p = .017; treatment d = 0.14, control d = 0.10, so discernment was 1.4 times higher in treatment versus control, about 50% attenuation from the target study’s 2.8 times higher effect.
  • Treatment-group participants were significantly less willing to share false headlines (M = 2.89 vs. control M = 3.04, p = .035, d = 0.11), but there was no significant difference for real headlines (M = 3.15 vs. 3.23, p = .27, d = 0.06).
  • Perceived headline accuracy correlated with treatment impact (r = .57, p = .001), supporting the target study’s proposed mechanism; the nudge was significant for Democrats (p = .016) but not Republicans (p = .315), and for non-Trump voters (p = .003) but not Trump voters (p = .653), though the relevant three-way interactions were not significant.

Methods

  • Preregistered direct replication under the COS SCORE program, using a two-stage design: Stage 1 recruited 701 participants for 90% power to detect 75% of the original effect; after failure, Stage 2 added 882 participants for a pooled sample powered at 90% to detect 50% of the original effect.
  • Participants were U.S. quota samples matched on age, gender, ethnic background, and geographic region, recruited via Respondi; treatment participants first rated the accuracy of one unrelated headline, then both groups rated 15 real and 15 false COVID-19 headlines on 6-point sharing-intention scales.
  • Main analysis used linear regression with robust standard errors clustered on participants and headlines (ivreg2; reghdfe robustness check), plus independent-samples and Bayesian t tests for false headlines, real headlines, and discernment; exploratory analyses examined headline order, attentiveness, partisanship, CRT, science knowledge, and MMS.

Limitations

  • The paper states that Stage 1 was unsuccessful, the pooled effect was attenuated by about 50%, and possible explanations include later pandemic timing, a changed headline set, and use of Respondi rather than Lucid or Mechanical Turk.
  • The paper states that the decay analysis was exploratory and used headline display order as a proxy because Qualtrics did not export elapsed time; it also notes no preregistered correction for the two-stage data-collection process.
  • The paper notes that attentive participants were less responsive to the nudge, though the three-way interaction with attentiveness was not significant, and it discusses an unreported change in the target study’s hypothesis from the preregistration; evident from the text, the sample was a non-probability quota sample and attention-check failures were not excluded from main analyses.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/09567976211024535

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In Stage 1 (n = 701), the interaction between headline veracity and treatment on sharing intentions was not statistically significant (β = 0.0046, 95% CI [−0.016, 0.026], p = .67).

    “This analysis yielded no significant interaction between headline veracity and treatment, β = 0.0046, 95% confidence interval (CI) = [−0.016, 0.026], F(3, 21030) = 1.53, p = .67”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.
  • Findingown data

    Using high-dimensional fixed-effects linear regression (reghdfe), the interaction between headline veracity and treatment remained significant (β = 0.015, 95% CI [0.0020, 0.028], p = .026).

    “Doing so also yielded a significant (uncorrected) effect for the interaction of headline veracity and treatment, β = 0.015, 95% CI = [0.0020, 0.028], F(3, 47490) = 4.52, p = .026”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.
  • Findingown data

    In the pooled sample, the treatment group was significantly less willing to share false headlines than the control group (treatment M = 2.89, control M = 3.04, mean difference = 0.15, 95% CI [0.011, 0.29], t(1581) = 2.113, p = .035, d = 0.11).

    “We found that—unlike in the target study—an independent-samples t test showed a significant difference for average willingness to share false headlines, in the sense that the treatment group was significantly less willing to share false headlines than the control group (treatment: M = 2.89, control: M = 3.04, mean difference: = 0.15, 95% CI = [0.011, 0.29]), t(1581) = 2.113, p = .035, d = 0.11, 95”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.
  • Findingown data

    In the pooled sample (N = 1,583), there was a small but significant interaction between treatment condition and headline veracity on sharing intentions (β = 0.015, 95% CI [0.0027, 0.027], p = .017).

    “This analysis yielded a significant interaction effect between headline veracity and treatment, β = 0.015, 95% CI = [0.0027, 0.027], F(3, 47490) = 4.52, p = .017”
    • Also stated in: § Abstract · plain
      “After collecting a second round of data (additional n = 882, pooled N = 1,583), we found a small but significant interaction between treatment condition and truth discernment (uncorrected p = .017; treatment: d = 0.14, control: d = 0.10).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.
  • Findingown data

    In the pooled sample, there was no significant difference between treatment and control groups in willingness to share real headlines (treatment M = 3.15, control M = 3.23, mean difference = 0.075, 95% CI [−0.057, 0.21], t(1581) = 1.110, p = .27, d = 0.06).

    “Also unlike the target study and against our Hypothesis 2, our analyses found no significant difference for real headlines (treatment: M = 3.15, control: M = 3.23, mean difference: 0.075, 95% CI = [−0.057, 0.21]), t(1581) = 1.110, p = .27, d = 0.06, 95% CI = [−0.043, 0.15].”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.
  • May combine two claims. 12
  • Classified as probably not a substantial claim. 15

  • May combine two claims.

    Findingown data

    In the pooled sample, the treatment-group effect size for discernment was d = −0.14 (95% CI [−0.17, −0.12]) and the control-group effect size was d = −0.10 (95% CI [−0.13, −0.078]).

    “treatment-group effect size: d = −0.14, 95% CI = [−0.17, −0.12] (see Table S6 in the Supplemental Material). The effect size for the control group was directionally similar (d = −0.10, 95% CI = [−0.13, −0.078])”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.

  • May combine two claims.

    Findingown data

    A Bayesian t test for real headlines yielded BF10 = 0.732, providing weak support against Hypothesis 2, with treatment-group participants indicating less willingness to share real news headlines than control-group participants.

    “A Bayesian t test provided weak support against Hypothesis 2; treatment-group participants indicated being less willing to share real news headlines than control-group participants (BF10 = 0.732, M = 0.028, 95% CI = [−0.040, 0.12], error percentage = 5.15 × 10−7”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.

  • May combine two claims.

    Findingown data

    In the second-stage replication, the treatment effect was just 1.4 times higher (treatment: d = 0.14, control: d = 0.10), about 50% attenuation relative to the original study.

    “there was about 50% attenuation in our second-stage replication, so the treatment effect was just 1.4 times higher (treatment: d = 0.14, control: d = 0.10).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The preregistration for this replication project was coordinated and verified according to COS protocols as part of the SCORE program.

    “The preregistration for this replication project was coordinated and verified according to COS protocols as part of the SCORE program and can be found on OSF (https://osf.io/rkfq5/).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In Pennycook et al. (2020), the truth-discernment finding was driven by an increase in sharing real news headlines rather than a reduction in sharing false headlines; the intervention did not reduce sharing of false headlines.

    “Importantly, the truth-discernment finding reported in the published article was actually driven by a different pattern of headline ratings: There was an increase in sharing real news headlines (as opposed to a reduction in sharing false headlines). As reported, the intervention did not reduce sharing of false headlines”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pennycook et al., 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The Stage 1 null interaction was robust to using high-dimensional fixed-effects linear regression (reghdfe) instead of ivreg2.

    “In addition, to ensure this was not due to minor differences in the clustering algorithm, we ran the same analysis using STATA’s reghdfe package (which uses high-dimensional fixed-effects linear regression; Correia, 2016), and these also yielded no significant interaction between headline veracity and treatment, β = 0.0046, 95% CI = [−0.018, 0.027], F(3, 21030) = 1.53, p = .68”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The target study used a sample matched to the U.S. population on age, gender, ethnicity, and geographic region but not obtained via probability sampling.

    “The target study by Pennycook et al. (2020) used a sample that was “matched to the U.S. population on age, gender, ethnicity, and [geographic] region” but was “not obtained via probability sampling and therefore should not be considered truly nationally representative” (p. 778).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Pennycook et al., 2020

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Pennycook et al. (2020) found that asking people to think about the accuracy of a single headline improves truth discernment of intentions to share news headlines about COVID-19.

    “the present study consisted of a two-stage replication test of a central finding by Pennycook et al. (2020), namely that asking people to think about the accuracy of a single headline improves “truth discernment” of intentions to share news headlines about COVID-19.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pennycook et al., 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    This high-powered, preregistered replication offers nuance to the existing body of research on accuracy nudges.

    “This high-powered, preregistered replication of Pennycook et al.’s study offers nuance to the existing body of research on accuracy nudges.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.
    • Rests on an assumption: The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.
    • Rests on an assumption: The Respondi quota sample is comparable to the target study's sample for the purpose of replication.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    No correction for the two-stage data-collection process was applied to control the family-wise error rate.

    “We did not preregister a correction for the two-stage data-collection process, so we present results here for the pooled sample according to our preregistration (i.e., without applying a correction to control the family-wise error rate).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The pooled sample was powered to achieve 90% power to detect 50% of the original effect size, which required an additional 882 participants for a total of 1,583.

    “The pooled sample (both the first and second stage) must achieve 90% power to detect 50% of the original effect size; in our case, this meant that an additional 882 participants (or 26,443 ratings) would be recruited if the replication failed in the first round, for a total of 1,583 participants in the pooled sample.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Independent reviewers identified by COS as part of the SCORE program evaluated the detailed preregistration.

    “To ensure that our study would be an honest and fair attempt to replicate the original findings, independent reviewers identified by COS as part of the SCORE program evaluated our detailed preregistration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The replication collected national quota samples matched to the U.S. population on age, gender, ethnic background, and geographic region from Respondi.

    “For this replication, we also opted to collect national quota samples that were matched to the U.S. population on age, gender, ethnic background, and geographic region but from a different platform, namely Respondi, a panel provider that is certified by the International Organization for Standardization (respondi.com).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Three attention checks were included, and participants who failed them were not excluded from analysis.

    “Following the target study, we included three attention checks in the form of subtle instructions in the middle of question blocks (Berinsky et al., 2014). Following Pennycook et al. (2020) and our preregistration, we did not exclude participants who failed attention checks from analysis, but we report results for different levels of attentiveness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    A direct consequence of the inattention-based account is that sharing behavior is amenable to accuracy interventions.

    “On an applied level, these results are also relevant because a direct consequence of the inattention-based account is that sharing behavior is amenable to accuracy interventions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Caution is advised when applying behavioral-science research directly to policy, particularly during the COVID-19 pandemic, because evidence readiness is not always clear.

    “However, as IJzerman et al. (2020) note, caution is advised when applying behavioral-science research directly to policy (particularly during the COVID-19 pandemic), as the evidence readiness of research findings is not always clear.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites IJzerman et al., 2020

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The inattention-based account of misinformation sharing directly challenges a motivational account of why people share fake news.

    “The findings of this study have high theoretical relevance in that the authors are advancing a largely cognitive-inattention-based account of misinformation sharing that directly challenges a motivational account of why people share fake news (Pennycook et al., 2021; Pennycook & Rand, 2019).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Pennycook et al., 2021

      Not checked in the free run.

    • cites Pennycook & Rand, 2019

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous research has generally found that priming effects dissipate rapidly, within a span of seconds.

    “previous research has generally found that priming effects dissipate rapidly, within a span of seconds (Branigan et al., 1999; Trammell & Valdes, 1992)”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Branigan et al., 1999

      Not checked in the free run.

    • cites Trammell & Valdes, 1992

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    People tend to share false information about COVID-19 on social media more because they fail to think about accuracy rather than because they have an a priori motivation to share content consistent with their beliefs.

    “In other words, the authors argue that people tend to share false information about COVID-19 on social media more because they fail to think about accuracy rather than because they have an a priori motivation to share content that is consistent with their beliefs (Van Bavel & Pereira, 2018).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Van Bavel & Pereira, 2018

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The first stage of data collection was calculated to achieve 90% power to detect 75% of the target study's effect size.

    “the first stage of data collection, calculated to achieve 90% power to detect 75% of the target study’s effect size, was unsuccessful”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    It is difficult to predict how harmful a given piece of misinformation is going to be, and individual viral news stories can cause significant harm.

    “It is difficult to predict how harmful a given piece of misinformation is going to be, and individual viral news stories can cause significant harm (van der Linden & Roozenbeek, 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites van der Linden & Roozenbeek, 2020

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The pattern reported by Pennycook et al. (2020) was via a difference in sharing of real headlines, which is different from the hypothesis originally preregistered by the authors (via a difference in sharing of false headlines).

    “the pattern reported by Pennycook et al. (2020), namely, via a difference in sharing of real headlines, is different from the hypothesis originally preregistered by the authors (i.e., via a difference in sharing of false headlines).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pennycook et al., 2020

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Our findings are in partial agreement with those of the target study and with Pennycook et al. (2021).

    “Our findings are thus in partial agreement with those of the target study, as well as with a recent study by the same authors that found significant effects of accuracy nudges on sharing intentions in three separate experiments, including a Twitter study (Pennycook et al., 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Pennycook et al., 2020

      Not checked in the free run.

    • cites Pennycook et al., 2021

      Not checked in the free run.

23 of 23

  • The paper relies on this without support.

    The new headlines used in the replication are equivalent to the original headlines in relevant properties such as veracity, plausibility, and topic.

    “Some of the true and false headlines about COVID-19 that were used originally were outdated when we conducted our replication. We therefore selected a series of new headlines to use in our study, which were kindly provided and approved by the original authors.”

  • The paper relies on this without support.

    The accuracy-rating task successfully induced participants in the treatment group to think about accuracy.

    “Before starting the task, treatment-group participants rated the accuracy of a single headline that was unrelated to COVID-19. Following the target study, we framed this as being part of a pretest.”

  • The paper relies on this without support.

    The Respondi quota sample is comparable to the target study's sample for the purpose of replication.

    “For this replication, we also opted to collect national quota samples that were matched to the U.S. population on age, gender, ethnic background, and geographic region but from a different platform, namely Respondi”

  • The authors state this as an assumption.

    The display order of headlines is a useful proxy for the time elapsed between exposure to the accuracy nudge and rating an individual headline.

    “We therefore considered looking at the display order of headlines (which were shown in a random order) to be a useful proxy for the time between being exposed to the accuracy nudge and rating an individual headline.”

  • The paper relies on this without support.

    Perceived headline accuracy from the original article's Study 1 is applicable to the headlines used in this replication.

    “perceived headline accuracy (from the original article’s Study 1, in which participants were asked to indicate the accuracy of all 30 headlines) was correlated with the impact of the accuracy nudge on sharing discernment.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Experimental work indicates that directing users' attention to accuracy can improve sharing quality and reduce misinformation circulation.

    “Experimental work indicates that directing users' attention to accuracy can improve sharing quality and reduce misinformation circulation (Pennycook et al., 2020(Pennycook et al., , 2021;;Roozenbeek et al., 2021).”

    From Sensational Headlines, Engagement, and Credibility in Nepal’s Online News (2025) · cites this paper as Roozenbeek et al., 2021

    The Results (Stage 2; page unavailable) report improved truth discernment and significantly lower willingness to share false headlines after an accuracy nudge. This supports improved sharing quality and a possible reduction in misinformation sharing, but the experiment measured stated sharing intentions, not whether misinformation circulation actually fell. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Accuracy prompts can modestly improve users' sharing decisions.

    “Pennycook et al. (2020) and Roozenbeek et al. (2021) show that accuracy prompts can modestly improve users' sharing decisions, while Pennycook et al. (2021) argue that shifting attention toward accuracy can reduce misinformation sharing online.”

    From Sensational Headlines, Engagement, and Credibility in Nepal’s Online News (2025) · cites this paper as Roozenbeek et al. (2021)

  • The accuracy prompt effect has been replicated in pre-registered studies conducted by other research groups.

    “This effect has been replicated in pre-registered studies conducted by other research groups29”

    From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [29]

  • The one replication of an accuracy prompt intervention that meets inclusion criteria (conducted in the U.S. between 2017 and 2020) did find a significant positive effect on sharing discernment.

    “the one replication of an accuracy prompt intervention29 we are aware of that meets our inclusion criteria (conducted in the U.S. between 2017 and 2020) did find a significant positive effect on sharing discernment”

    From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [29]

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Not confirmedF(3, 21030) = 1.53reported p = .67 · recomputed p = .204

    “F(3, 21030) = 1.53, p = .67”

  • Not confirmedF(3, 21030) = 1.53reported p = .68 · recomputed p = .204

    “F(3, 21030) = 1.53, p = .68”

  • Not confirmedF(3, 47490) = 4.52reported p = .026 · recomputed p = .004

    “F(3, 47490) = 4.52, p = .026”

  • Not confirmedF(7, 47490) = 4.49reported p = .052 · recomputed p < .001

    “F(7, 47490) = 4.49, p = .052”

  • Not confirmedF(3, 36930) = 3.66reported p = .065 · recomputed p = .012

    “F(3, 36930) = 3.66, p = .065”

  • Not confirmedF(3, 13290) = 3.87reported p = .64 · recomputed p = .009

    “F(3, 13290) = 3.87, p = .64”

  • Not confirmedF(7, 47490) = 17.91reported p = .337 · recomputed p < .001

    “F(7, 47490) = 17.91, p = .337”

  • Not confirmedF(3, 32910) = 6.74reported p = .016 · recomputed p < .001

    “F(3, 32910) = 6.74, p = .016”

  • Not confirmedF(3, 14580) = 0.36reported p = .315 · recomputed p = .782

    “F(3, 14580) = 0.36, p = .315”

  • Not confirmedF(3, 32490) = 6.58reported p = .003 · recomputed p < .001

    “F(3, 32490) = 6.58, p = .003”

  • Not confirmedF(3, 15000) = 0.12reported p = .653 · recomputed p = .948

    “F(3, 15000) = 0.12, p = .653”

  • ConsistentF(3, 47490) = 4.52reported p = .017 · recomputed p = .004

    “F(3, 47490) = 4.52, p = .017”

    The paper states a correction for multiple tests

  • Consistentt(1581) = -2.013reported p = .044 · recomputed p = .044

    “t(1581) = −2.013, p = .044”

  • Consistentt(1581) = 2.113reported p = .035 · recomputed p = .035

    “t(1581) = 2.113, p = .035”

  • Consistentt(1581) = 1.11reported p = .27 · recomputed p = .267

    “t(1581) = 1.110, p = .27”

15 of 15

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Headline-order confidence interval conflicts with its p-value

    Major

    The reported 95% confidence interval excludes zero, whereas the reported two-sided p-value of .052 exceeds .05. These give conflicting significance results for the same interaction.

  • Trump-voting subgroup confidence interval excludes its estimate

    Minor

    The reported estimate, 0.0231, falls outside both endpoints of its stated confidence interval. The interval's endpoints are also printed in descending order.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Correia S., 2016, Stata Conference 2016

Cited by

Sorted by citations

Related

Authors
Roozenbeek, J., Freeman, A. L. J., & van der Linden, S.
Affiliations
  • University of Cambridge (GB)
  • Department of Psychology, University of Cambridge
ORCID
Venue
Psychological Science
Publisher
SAGE Publications
Published
2021-06-11
Volume
32
Issue
7
Pages
1169-1178
ISSN
  • 0956-7976 (ISSN-L)
  • 1467-9280
DOI
10.1177/09567976211024535
PMC
PMC8641132
OpenAlex
W3172358054
Semantic Scholar
e773b41de41b65820e24236c92e6d51b094570c4
Cited by
121
References
33
Citations by year
2026: 10 · 2025: 11 · 2024: 24 · 2023: 37 · 2022: 30 · 2021: 8 · 2020: 1
FWCI
34.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Psychological Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Apollo (University of Cambridge) (submittedVersion)
  • Apollo
Funders
  • Defense Advanced Research Projects Agency
  • Winton Centre for Risk & Evidence Communication
  • David & Claudia Harding Foundation
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Communication › Communication and COVID-19 Impact
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
truth discernment, misinformation, direct replication, preregistration
MeSH terms
COVID-19, Humans, Mass Media, Thinking, Truth Disclosure, Reproducibility of Results, Information Dissemination, Intention

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review