CoChat
Sign Up
Cited by 272Misinformation and Its ImpactsAcademic Publishing and Open AccessCommunication and COVID-19 Impact

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Open access
Pennycook, Gordon, Rand, David G.·2022·Nature Communications
DOI 10.1038/s41467-022-30073-5PMC PMC9051116
Open this paper in CoChat — free accountOpen PDF

Abstract

Interventions that shift users attention toward the concept of accuracy represent a promising approach for reducing misinformation sharing online. We assess the replicability and generalizability of this accuracy prompt effect by meta-analyzing 20 experiments (with a total N = 26,863) completed by our group between 2017 and 2020. This internal meta-analysis includes all relevant studies regardless of outcome and uses identical analyses across all studies. Overall, accuracy prompts increased the quality of news that people share (sharing discernment) relative to control, primarily by reducing sharing intentions for false headlines by 10% relative to control in these studies. The magnitude of the effect did not significantly differ by content of headlines (politics compared with COVID-19 related news) and did not significantly decay over successive trials. The effect was not robustly moderated by gender, race, political ideology, education, or value explicitly placed on accuracy, but was significantly larger for older, more reflective, and more attentive participants. This internal meta-analysis demonstrates the replicability and generalizability of the accuracy prompt effect on sharing discernment.

AI snapshot

Written from the paper’s full text.

TL;DR

This internal meta-analysis of 20 experiments (total N = 26,863) finds that accuracy prompts reliably increase sharing discernment, mainly by reducing intentions to share false news by about 10% relative to control. The authors conclude the effect is replicable and generalizable across prompt types, topics, and samples, though effects are modest and mostly tested in U.S. online survey experiments by one research group.

Key findings

  • Across all 20 experiments, accuracy prompts significantly increased sharing discernment (interaction between headline veracity and treatment: b = 0.038, z = 7.102, p < 0.001), a 71.7% increase over baseline control discernment. This was driven by a significant decrease in sharing intentions for false news (b = −0.034, z = 7.851, p < 0.001), a 10% decrease relative to control, with no significant effect on true news (b = 0.006, z = 1.44, p = 0.150).
  • The most-tested Evaluation prompt (k = 14) significantly increased discernment (b = 0.034, p < 0.001), a 59.6% increase over baseline, and reduced false-news sharing by 8.2%. Combined Evaluation-plus-additional treatments were roughly twice as effective as Evaluation alone (discernment b = 0.054, p = 0.006; false-news sharing decrease 16.5%), and non-Evaluation treatments also increased discernment (b = 0.039, p < 0.001; false-news sharing decrease 11.0%).
  • Effects did not significantly differ for political versus COVID-19 headlines (b = −0.017, p = 0.244) and did not significantly decay over successive trials (three-way interaction with trial number: b = −0.001, p = 0.062; excluding first four trials: b = −0.000, p = 0.770).
  • Abstract reports no robust moderation by gender, race, political ideology, education, or value explicitly placed on accuracy, but effects were significantly larger for older, more reflective, and more attentive participants. At the item level, the treatment effect was strongly correlated with a headline’s out-of-sample perceived accuracy (meta-analytic r = 0.773, z = 19.417, p < 0.001; pooled r(355) = 0.663, p < 0.001).

Methods

  • Design: internal meta-analysis of 20 experiments conducted by the authors’ group between 2017 and 2020, including all qualifying studies regardless of outcome and using identical analyses across studies.
  • Data/sample: U.S. online survey participants (N = 26,863) recruited from Amazon Mechanical Turk, Lucid, or YouGov; participants indicated sharing intentions for true and false headlines about politics and/or COVID-19 after randomization to control or an accuracy prompt.
  • Analysis: rating-level linear regressions with robust standard errors clustered on participant and headline, with sharing intentions rescaled to [0,1]; study-level coefficients were combined using random effects meta-analysis. Main specification included headline veracity, treatment condition, and their interaction.

Limitations

  • Stated: the internal meta-analysis does not take external studies into account; only one external replication meeting inclusion criteria was included, and future work should examine studies from a wider range of research groups, across cultures, and in more field experiments. The authors also note the effects are modest in size, attention was measured post-treatment in the attentiveness analyses, and power for heterogeneity analyses across studies was comparatively low.
  • Stated: MTurk samples are not nationally representative and can produce misleading individual-difference results, particularly related to politics; the authors privilege Lucid/YouGov results for individual-level moderation.
  • Evident from the text: all included experiments came from the authors’ own group and used U.S. samples; the meta-analysis itself was not pre-registered and no protocol was prepared; Table 3 notes no adjustments were made for multiple comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-022-30073-5

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

0 of 51 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The accuracy prompt effect was significantly larger for more reflective participants.

    “but was significantly larger for older, more reflective, and more attentive participants.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The accuracy prompt effect was not robustly moderated by the value explicitly placed on accuracy.

    “The effect was not robustly moderated by gender, race, political ideology, education, or value explicitly placed on accuracy”
    Quote in the paperSubstantial claim
  • Findingown data

    The magnitude of the accuracy prompt effect did not significantly differ by headline content (politics compared with COVID-19 related news).

    “The magnitude of the effect did not significantly differ by content of headlines (politics compared with COVID-19 related news)”
    • Also stated in: § Study-level differences · plain
      “There were, however, no significant differences in the effect for political relative to COVID-19 headlines, b = −0.017, t = −1.21, p = 0.244.”
    Quote in the paperSubstantial claim
  • Findingown data

    The accuracy prompt effect was not significantly moderated by gender in either the more representative samples or MTurk.

    “With respect to gender, we find no significant moderation in either the more representative samples or MTurk.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The accuracy prompt effect did not significantly decay over successive trials.

    “and did not significantly decay over successive trials.”
    • Also stated in: § Item-level differences · hedged
      “Examining the three-way interaction between headline veracity, treatment dummy, and trial number, the meta-analytic estimate (with a total of N = 10,236 subjects) is not statistically significant, b = −0.001, z = −1.869, p = 0.062.”
    Quote in the paperSubstantial claim
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 21

  • May combine two claims.

    Findingown data

    The accuracy prompt effect was significantly larger for older participants in the more representative samples but not MTurk.

    “With respect to age, we find that the accuracy prompts have a significantly larger effect for older participants in the more representative samples but not MTurk”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The accuracy prompt effect was not significantly moderated by race/ethnicity (white versus other categories) in either subject pool, and there were no significant differences in baseline discernment.

    “With respect to participants who identified as white versus other ethnicity or racial categories, we also find no significant moderation in either set of subject pools, and no significant differences in baseline discernment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Accuracy prompts significantly increased sharing discernment relative to control (interaction b = 0.038, z = 7.102, p < 0.001), translating into a 71.7% increase over baseline sharing discernment.

    “Overall, accuracy prompts increased the quality of news that people share (sharing discernment) relative to control”
    • Also stated in: § Results · plain
      “We find that accuracy prompts significantly increase sharing discernment (interaction between headline veracity and treatment dummies: b = 0.038, z = 7.102, p < 0.001; Fig. 1), which translates into a 71.7% increase over the meta-analytic estimate of baseline sharing discernment in the control condition”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Accuracy prompts significantly decreased sharing intentions for false news (treatment dummy b = −0.034, z = 7.851, p < 0.001), translating into a 10% decrease relative to baseline sharing intentions for false news.

    “primarily by reducing sharing intentions for false headlines by 10% relative to control in these studies.”
    • Also stated in: § Results · plain
      “This increase in discernment was driven by accuracy prompts significantly decreasing sharing intentions for false news (treatment dummy: b = −0.034, z = 7.851, p < 0.001; Fig. 2), which translates into a 10% decrease relative to the meta-analytic estimate of baseline sharing intentions for false news in the control condition”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The non-evaluation treatments significantly decreased sharing intentions for false news (b = −0.039, z = −5.161, p < 0.001), translating into an 11.0% decrease relative to baseline.

    “This increase in discernment was again largely driven by a significant decrease in sharing intentions for false news, b = −0.039, z = −5.161, p < 0.001, which translates into a 11.0% decrease relative to baseline sharing intentions for false news in the control condition of those experiments”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The non-evaluation treatments significantly increased sharing discernment (b = 0.039, z = 4.974, p < 0.001), translating into a 70.9% increase over baseline.

    “The non-evaluation treatments significantly increased sharing discernment (interaction between headline veracity and treatment dummies: b = 0.039, z = 4.974, p < 0.001), which translates into a 70.9% increase over baseline sharing discernment in the control condition of those experiments”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Evaluation treatment significantly decreased sharing intentions for false news (b = −0.027, z = −5.548, p < 0.001), translating into an 8.2% decrease relative to baseline.

    “This increase in discernment was again largely driven by the Evaluation treatment significantly decreasing sharing intentions for false news (treatment dummy: b = −0.027, z = −5.548, p < 0.001), which translates into an 8.2% decrease relative to baseline sharing intentions for false news in the control condition”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The meta-analysis includes k = 20 experiments with a total of N = 26,863 participants.

    “In total, we meta-analyze k = 20 experiments, with a total of N = 26,863 participants (see Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the more representative samples, across three partisanship measures, Republicans were significantly less discerning in baseline sharing, but partisanship did not significantly moderate the accuracy prompt effect.

    “In the more representative samples, across all three measures we find that people who identify as Republican were significantly less discerning in their baseline sharing (in line with what has been observed previously), but partisanship does not significantly moderate the accuracy prompt effect on discernment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Women were significantly less discerning in the control condition on MTurk, but not in the more representative samples.

    “Interestingly, women are significantly less discerning in the control condition on MTurk, but not in the more representative samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On MTurk, Republicans were not significantly less discerning in baseline sharing, yet partisanship did significantly moderate the accuracy prompt effect, with prompts working less well for Republicans.

    “On MTurk, we see the opposite pattern: Republicans were not significantly less discerning in their baseline sharing, yet partisanship does significantly moderate the accuracy prompt on discernment, with the prompts working less well for Republicans”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In re-analysis of the Twitter field experiment, more conservative users shared lower quality information at baseline, but ideology did not moderate the effect of accuracy prompts.

    “Critically, the pattern of results for this Twitter data matches what we report above for the survey experiments: Users who are more conservative shared lower quality information at baseline, but we do not find evidence that ideology moderated the effect of the accuracy prompts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There was a significant negative relationship between conservatism and baseline sharing discernment in the control in both subject pools, with the relationship twice as large in the more representative samples.

    “Importantly, we do find a significant negative relationship between conservatism and baseline sharing discernment (i.e., overall quality of news that is shared) in the control, and this is evident in both sets of subject pools (although the relationship is twice as large in the more representative samples).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The treatment effect is strongly correlated with perceived accuracy in all experiments; the meta-analytic estimate of the correlation is r = 0.773, z = 19.417, p < 0.001.

    “Consistent with our proposed mechanism, the treatment effect is strongly correlated with perceived accuracy in all experiments (Fig. 4): the meta-analytic estimate of the correlation is r = 0.773, z = 19.417, p < 0.001”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Out-of-sample average accuracy ratings of headlines are valid measures of participants’ perceived accuracy.

  • May combine two claims.

    Findingown data

    Across 20 experiments with U.S. samples, a variety of accuracy prompts robustly increased social media users' sharing discernment by decreasing their intentions to share a wide range of false news headlines about politics and COVID-19.

    “Here, we provide evidence, across 20 experiments with U.S. samples, that a variety of accuracy prompts robustly increased social media users’ sharing discernment by decreasing their intentions to share a wide range of false news headlines about politics and COVID-19.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' self-reported likelihood of sharing headlines is a valid proxy for their actual sharing behavior on social media.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Including this replication study in the main analysis does not meaningfully alter the results.

    “including this study in our main analysis does not meaningfully alter the results (see SI Section 4)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-reported likelihood of sharing headlines is a valid proxy for their actual sharing behavior on social media.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Accuracy prompts do not require stakeholders such as technology companies or government regulators to decide what is true versus false.

    “Another advantage is that accuracy prompts do not require stakeholders such as technology companies or government regulators to decide what is true versus false.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Prompting participants to think about accuracy reduces the disconnect between accuracy judgments and sharing intentions and thereby increases the quality of news shared.

    “Evidence for the role of inattention comes from experiments in which prompting participants to think about the concept of accuracy – for example, by asking them to evaluate the accuracy of a random headline at the beginning of the study – reduces the disconnect between accuracy judgments and sharing intentions, and thereby increases the quality of news shared24–26.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

      Not checked in the free run.

    • cites [25] Epstein, Z. et al. Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online. Harvard Kennedy Sch. Misinformation Rev. https://doi.org/10.37016/mr-2020-71 (2021). source

      Not checked in the free run.

    • cites [26] Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention. Psychol. Sci. 31, 770–780. https://doi.org/10.31234/OSF.IO/UHBK9 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An overwhelming majority of participants say it is important to only share accurate news.

    “Ironically, this occurs despite the fact that an overwhelming majority of participants say that it is important to only share accurate news24.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    Study selection was broad and not susceptible to motivated choices about inclusion.

    “As a result, study selection was broad and not susceptible to motivated choices about inclusion.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The meta-analysis included all interventions that could be construed as accuracy prompts: interventions that occurred prior to the sharing task, invoked accuracy, and did not provide specific information about the veracity of particular headlines.

    “Furthermore, we included all interventions that could be construed as accuracy prompts – i.e., interventions that occurred prior to the sharing task, that invoked accuracy in some way (such that the concept of accuracy would be primed), and did not provide any specific information about the veracity of any particular headlines (were “content neutral”).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Accuracy prompts leverage users' own abilities to identify misinformation and take advantage of the fact that most people want to share accurate content.

    “Instead, they leverage users’ own abilities to identify misinformation and take advantage of the fact that most people want to share accurate content24.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Sharing discernment is superior to simply focusing on sharing of false news because an intervention that decreases sharing of both true and false news equally would not indicate that people are focusing more on accuracy.

    “This approach is superior to simply focusing on the sharing of false news because an intervention that decreases the sharing of both true and false news equally would not indicate that people are focusing more on accuracy40.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Pennycook, G. & Rand, D. G. Lack of partisan bias in the identification of fake (versus real) news. Trends Cogn. Sci. 25, 725–726 (2021). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The combined treatments were roughly twice as effective as Evaluation alone, suggesting substantial gains from stacking accuracy prompt interventions.

    “Although not a perfectly controlled comparison, the observation that the combined treatments were roughly twice as effective as Evaluation alone suggests that there are substantial gains from stacking accuracy prompt interventions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Although the meta-analysis was not itself pre-registered, the decision about what to include was made before conducting the meta-analysis.

    “Importantly, although our meta-analysis was not itself pre-registered, we made the decision about what to include before conducting the meta-analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    There are two main threats to the validity of meta-analytic results: systematic omission of studies and flexible selection of analysis approaches within each study.

    “There are two main threats to the validity of meta-analytic results: the systematic omission of studies (e.g., publication bias suppressing studies with null results36,37), and the flexible selection of analysis approaches within each study inflating the rate of false positives (e.g., p-hacking38).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [36] Sutton, A. J., Song, F., Gilbody, S. M. & Abrams, K. R. Modelling publication bias in meta-analysis: A review. Stat. Methods Med. Res. 9, 421–445 (2000). source

      Not checked in the free run.

    • cites [37] Stanley, T. D. Limitations of PET-PEESE and other meta-analysis methods. Soc. Psychol. Personal. Sci. 8, 581–591 (2017). source

      Not checked in the free run.

    • cites [38] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The worse baseline discernment among conservatives aligns with real-world sharing data where conservatives shared more fake news on Twitter and Facebook.

    “This worse baseline discernment aligns with real-world sharing data where conservatives were found to share more fake news on Twitter10 and Facebook12.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [10] Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B. & Lazer, D. Fake news on twitter during the 2016 U.S. Presidential election. Science 363, 374–378 (2019). source

      Not checked in the free run.

    • cites [12] Guess, A. M., Nagler, J. & Tucker, J. Less than you think: Prevalence and predictors of fake news dissemination on Facebook. Sci. Adv. 5, eaau4586 (2019). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The meta-analysis has comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference).

    “Although we have comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference),”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    We have comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference).

    “we have comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is a dissociation between accuracy judgments and sharing intentions observed in past work.

    “given the dissociation between accuracy judgments and sharing intentions observed in past work24–26”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

      Not checked in the free run.

    • cites [25] Epstein, Z. et al. Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online. Harvard Kennedy Sch. Misinformation Rev. https://doi.org/10.37016/mr-2020-71 (2021). source

      Not checked in the free run.

    • cites [26] Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention. Psychol. Sci. 31, 770–780. https://doi.org/10.31234/OSF.IO/UHBK9 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Attention was measured post-treatment, which has the potential to undermine inferences drawn from these data.

    “An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] Montgomery, J. M., Nyhan, B. & Torres, M. How conditioning on posttreatment variables can ruin your experiment and what to do about it. Am. J. Pol. Sci. 62, 760–775 (2018). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior work indicates that cognitive reflection is positively associated with truth discernment when making accuracy judgments.

    “Prior work indicates that cognitive reflection is positively associated with truth discernment when making accuracy judgments18,45”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [18] Pennycook, G. & Rand, D. G. The psychology of fake news. Trends Cogn. Sci. 25, 388–402 https://doi.org/10.1016/j.tics.2021.02.007 (2021). source

      Not checked in the free run.

    • cites [45] Pennycook, G. & Rand, D. G. Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition 188, 39–50 (2019). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    MTurk offers convenience samples and makes no attempt to be nationally representative.

    “MTurk offers convenience samples and makes no attempt to be nationally representative”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [51] Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J. & Litman, L. Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behav. Res. Methods 51, 2022–2038 (2019). source

      Not checked in the free run.

    • cites [52] Boas, T. C., Christenson, D. P. & Glick, D. M. Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics. Polit. Sci. Res. Methods 8, 232–250 (2020). source

      Not checked in the free run.

    • cites [53] Berinsky, A. J., Huber, G. A. & Lenz, G. S. Evaluating online labor markets for experimental research: amazon.com’s mechanical turk. Polit. Anal. 20, 351–368 (2012). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    If a group of participants is targeted who have a very difficult time distinguishing between true and false content, or if misinformation is identified that the majority of a particular group of people believe to be true, then accuracy prompts will not improve sharing discernment.

    “Thus, if a group of participants is targeted who have a very difficult time distinguishing between true and false content, or if misinformation is identified that the majority of a particular group of people believe to be true, then accuracy prompts will not improve sharing discernment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' self-reported likelihood of sharing headlines is a valid proxy for their actual sharing behavior on social media.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In most cases, headlines were selected for inclusion based on prior pre-testing; for all experiments using political headlines, the headlines were balanced on partisan lean based on the pre-test results.

    “In most cases, headlines were selected for inclusion based on prior pre-testing; for all experiments using political headlines, the headlines were balanced on partisan lean based on the pre-test results”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The inclusion and exclusion criteria were determined prior to running the meta-analysis.

    “Although these inclusion and exclusion criteria were determined prior to running the meta-analysis”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The meta-analysis was not pre-registered and no protocol was prepared.

    “the meta-analysis was not pre-registered and no protocol was prepared”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    Scholars have argued that replicable priming effects are restricted to perceptual primes.

    “scholars have argued that replicable priming effects are restricted to “perceptual primes” (e.g., as in semantic priming), which involve (for example) perceiving an ambiguous stimuli in a way that is more consistent with a prime58”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [58] Harris, C., Rohrer, D. & Pashler, H. A train wreck by any other name. Psychological Inq. 32, 17–23 (2021). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    This form of priming is said to occur when a stimulus influences subsequent behavior without conscious guidance or intention.

    “This form of priming is said to occur when “a stimulus influences subsequent behavior without conscious guidance or intention”55”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Sherman, J. W. & Rivers, A. M. There’s nothing social about social priming: derailing the “Train Wreck”. Psychol. Inq. 32, 1–11 (2021). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The benefit of internal meta-analyses is that one can be confident that the entire file-drawer is included and that p-hacking has been avoided.

    “Although the benefit of internal meta-analyses is that one can be confident that the entire “file-drawer” is included and that p-hacking has been avoided”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Both authors agreed on the classification of each study as eligible versus ineligible.

    “Both authors agreed on the classification of each study as eligible versus ineligible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Many studies prevented respondents who failed trivial attention checks at the study outset from participating, but since no data was collected from these participants, we cannot assess the impact of extreme inattention.

    “Many studies also prevented respondents who failed trivial attention checks at the study outset from participating, but since no data was collected from these participants, we cannot assess the impact of extreme inattention.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several prominent examples of behavioral priming have proven difficult to replicate.

    “Several prominent examples of behavioral priming have proven difficult to replicate56,57”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [56] Cesario, J. Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9, 40–48 (2014). source

      Not checked in the free run.

    • cites [57] Molden, D. C. Understanding priming effects in social psychology: An overview and integration. Soc. Cognition 32, 243–249 (2014). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We use random effects meta-analysis, rather than fixed-effects meta-analysis, because there is reason to expect that the true effect size varies across studies.

    “We use random effects meta-analysis, rather than fixed-effects meta-analysis, because there is reason to expect that the true effect size varies across studies”
    Quote in the paperSubstantial claim
44 of 44

  • The paper relies on this without support.

    The massive networked character of social media platforms means that when people choose to share misinformation online, it has the potential to reach and influence a large number of others.

    “The massive networked character of social media platforms means that when people choose to share misinformation online, it has the potential to reach (and influence) a large number of others.”

  • The paper relies on this without support.

    Out-of-sample average accuracy ratings of headlines are valid measures of participants’ perceived accuracy.

    “for 15 experiments we have out-of-sample ratings of the average accuracy of each headline (e.g., from pre-tests, or from other experiments or conditions where participants made accuracy ratings rather than indicating sharing intentions).”

  • The paper relies on this without support.

    Participants' self-reported likelihood of sharing headlines is a valid proxy for their actual sharing behavior on social media.

    “participants were asked to indicate their likelihood of sharing a set of true and false headlines”
    “sharing decisions (the dependent variable) are rescaled such that the minimum possible value is 0 (indicating a very low likelihood of sharing) and the maximum possible value is 1 (indicating a very high likelihood of sharing)”

  • The paper relies on this without support.

    The professional fact-checker ratings of headline veracity are accurate.

    “the veracity of which were rated by professional fact-checkers”

  • The paper relies on this without support.

    The accuracy prompts used in the experiments are representative of accuracy prompts that could be deployed on social media platforms.

    “a variety of accuracy prompts”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = 7.102reported p < .001 · recomputed p < .001

    “z = 7.102, p < .001”

  • Consistentz = 6.636reported p < .001 · recomputed p < .001

    “z = 6.636, p < .001”

  • Consistentz = 7.851reported p < .001 · recomputed p < .001

    “z = 7.851, p < .001”

  • Consistentz = 15.695reported p < .001 · recomputed p < .001

    “z = 15.695, p < .001”

  • Consistentz = 1.44reported p = .15 · recomputed p = .150

    “z = 1.44, p = .150”

  • ConsistentQ(19) = 88.53reported p < .001 · recomputed p < .001

    “Q(19) = 88.53, p < .001”

  • ConsistentQ(19) = 23.33reported p = .223 · recomputed p = .223

    “Q(19) = 23.33, p = .223”

  • ConsistentQ(19) = 22.42reported p = .264 · recomputed p = .264

    “Q(19) = 22.42, p = .264”

  • Consistentz = 7.823reported p < .001 · recomputed p < .001

    “z = 7.823, p < .001”

  • Consistentz = 7.2reported p < .001 · recomputed p < .001

    “z = 7.2, p < .001”

  • Consistentz = -5.548reported p < .001 · recomputed p < .001

    “z = −5.548, p < .001”

  • Consistentz = 14.1reported p < .001 · recomputed p < .001

    “z = 14.1, p < .001”

  • Consistentz = 1.89reported p = .059 · recomputed p = .059

    “z = 1.89, p = .059”

  • Consistentz = 2.765reported p = .006 · recomputed p = .006

    “z = 2.765, p = .006”

  • Consistentz = 2.92reported p = .003 · recomputed p = .004

    “z = 2.92, p = .003”

  • Consistentz = -2.99reported p = .003 · recomputed p = .003

    “z = −2.990, p = .003”

  • Consistentz = 8.7reported p < .001 · recomputed p < .001

    “z = 8.7, p < .001”

  • Consistentz = 0.775reported p = .438 · recomputed p = .438

    “z = 0.775, p = .438”

  • Consistentz = 4.974reported p < .001 · recomputed p < .001

    “z = 4.974, p < .001”

  • Consistentz = 7.1reported p < .001 · recomputed p < .001

    “z = 7.1, p < .001”

  • Consistentz = -5.161reported p < .001 · recomputed p < .001

    “z = −5.161, p < .001”

  • Consistentz = 16.6reported p < .001 · recomputed p < .001

    “z = 16.6, p < .001”

  • Consistentz = 0.338reported p = .735 · recomputed p = .735

    “z = 0.338, p = .735”

  • Consistentz = 19.417reported p < .001 · recomputed p < .001

    “z = 19.417, p < .001”

  • ConsistentQ(14) = 18.99reported p = .165 · recomputed p = .165

    “Q(14) = 18.99, p = .165”

  • Consistentr(355) = 0.663reported p < .001 · recomputed p < .001

    “r(355) = 0.663, p < .001”

  • Consistentz = 3.124reported p = .002 · recomputed p = .002

    “z = 3.124, p = .002”

  • Consistentz = 11.276reported p < .001 · recomputed p < .001

    “z = 11.276, p < .001”

  • Consistentz = 1.085reported p = .278 · recomputed p = .278

    “z = 1.085, p = .278”

  • Consistentz = -1.869reported p = .062 · recomputed p = .062

    “z = −1.869, p = .062”

  • Consistentz = -0.292reported p = .77 · recomputed p = .770

    “z = −0.292, p = .770”

  • Consistentz = 4.188reported p < .001 · recomputed p < .001

    “z = 4.188, p < .001”

  • Consistentz = 4.804reported p < .001 · recomputed p < .001

    “z = 4.804, p < .001”

  • Consistentz = 5.707reported p < .001 · recomputed p < .001

    “z = 5.707, p < .001”

  • Consistentz = 4.94reported p < .001 · recomputed p < .001

    “z = 4.940, p < .001”

  • Consistentz = 3.404reported p = .001 · recomputed p < .001

    “z = 3.404, p = .001”

  • Consistentz = 2.988reported p = .003 · recomputed p = .003

    “z = 2.988, p = .003”

  • Consistentz = 2.79reported p = .005 · recomputed p = .005

    “z = 2.790, p = .005”

  • Consistentz = 2.608reported p = .009 · recomputed p = .009

    “z = 2.608, p = .009”

  • Consistentz = 2.784reported p = .005 · recomputed p = .005

    “z = 2.784, p = .005”

  • Consistentz = 2.119reported p = .034 · recomputed p = .034

    “z = 2.119, p = .034”

  • Consistentz = 3.581reported p < .001 · recomputed p < .001

    “z = 3.581, p < .001”

  • Consistentz = 0.726reported p = .468 · recomputed p = .468

    “z = 0.726, p = .468”

  • Consistentz = 4.44reported p < .001 · recomputed p < .001

    “z = 4.440, p < .001”

44 of 44

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Partisanship-measure study counts add to 20, not 18

    Minor

    The stated counts of studies using binary, six-point, and seven-point versions are 5, 13, and 2, respectively. These add to 20, although the measure is said to have been collected in 18 experiments.

    Reported 18 · recomputed 20

  • Combined-treatment increase does not match the stated baseline

    Minor

    An effect of 0.054 relative to the stated baseline discernment of 0.050 is a 108% increase, not 100.8%.

    Reported 100.8 · recomputed 108

  • Education-moderation claim contradicts Table 3

    Minor

    The text says accuracy prompts had a larger effect for college-educated participants in both sample types. Table 3 reports nonsignificant moderation in both, with a negative coefficient for MTurk.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Rivers, D. & Bailey, D. Inference from matched samples in the 2008 U.S. national elections. Proc. Jt. Stat. Meet. 627–639 (2009).

Cited by

Sorted by citations

Related

Authors
Pennycook, G., & Rand, D. G.
Affiliations
  • University of Regina (CA)
  • Massachusetts Institute of Technology (US)
Corresponding author
  • Pennycook, Gordon
  • Rand, David G.
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2022-04-28
Volume
13
Issue
1
Pages
2333-2333
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-022-30073-5
PMC
PMC9051116
OpenAlex
W4225127529
Cited by
270
References
74
Citations by year
2026: 66 · 2025: 59 · 2024: 76 · 2023: 41 · 2022: 23 · 2021: 2 · 2019: 2 · 2012: 1
FWCI
8.44
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Funders
  • Gouvernement du Canada | Canadian Institutes of Health Research
  • Gouvernement du Canada | Social Sciences and Humanities Research Council of Canada
  • William and Flora Hewlett Foundation
  • Google
  • National Science Foundation
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
  • Social Sciences › Communication › Communication and COVID-19 Impact
Keywords
misinformation, news sharing, replicability, generalizability, political ideology, COVID-19 news, political news
MeSH terms
COVID-19, Communication, Humans, Politics, Intention

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review