Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

In the paper

“MTurk offers convenience samples and makes no attempt to be nationally representative”

What the sources say

• Partly supported

  The discussion of MTurk’s limitations (page unavailable) says its workers look very little like the US population and identifies substantial demographic differences. The Method section describes recruiting available MTurk workers, consistent with a convenience sample. The paper does not establish the stronger claim that MTurk makes no attempt to be nationally representative. (LLM, conf=0.83)

  cites [51] Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J. & Litman, L. Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behav. Res. Methods 51, 2022–2038 (2019). sourceExperimental — accuracy not yet measured
• Supported

  The paper describes MTurk as a source of online convenience samples (Abstract and Conclusion). Its Demographics section compares the US MTurk sample with a probability sample and reports substantial demographic differences. This supports the paraphrase that MTurk recruitment does not aim to produce a nationally representative sample. (LLM, conf=0.88)

  cites [52] Boas, T. C., Christenson, D. P. & Glick, D. M. Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics. Polit. Sci. Res. Methods 8, 232–250 (2020). source
• Supported

  Section 4.1 describes recruiting the MTurk sample through a posted survey task rather than a nationally representative sampling procedure. The discussion in section 22 explicitly treats MTurk as a means of conducting experiments 'instead of a representative sample.' The authors also note that MTurk samples can be more diverse than other convenience samples; that comparison does not make them nationally representative. (LLM, conf=0.85)

  cites [53] Berinsky, A. J., Huber, G. A. & Lenz, G. S. Evaluating online labor markets for experimental research: amazon.com’s mechanical turk. Polit. Anal. 20, 351–368 (2012). source

In the paper

“With respect to age, we find that the accuracy prompts have a significantly larger effect for older participants in the more representative samples but not MTurk”

In the paper

“With respect to participants who identified as white versus other ethnicity or racial categories, we also find no significant moderation in either set of subject pools, and no significant differences in baseline discernment.”

In the paper

“Overall, accuracy prompts increased the quality of news that people share (sharing discernment) relative to control”

• Also stated in: § Results · plain

  “We find that accuracy prompts significantly increase sharing discernment (interaction between headline veracity and treatment dummies: b = 0.038, z = 7.102, p < 0.001; Fig. 1), which translates into a 71.7% increase over the meta-analytic estimate of baseline sharing discernment in the control condition”

In the paper

“primarily by reducing sharing intentions for false headlines by 10% relative to control in these studies.”

• Also stated in: § Results · plain

  “This increase in discernment was driven by accuracy prompts significantly decreasing sharing intentions for false news (treatment dummy: b = −0.034, z = 7.851, p < 0.001; Fig. 2), which translates into a 10% decrease relative to the meta-analytic estimate of baseline sharing intentions for false news in the control condition”

In the paper

“This increase in discernment was again largely driven by a significant decrease in sharing intentions for false news, b = −0.039, z = −5.161, p < 0.001, which translates into a 11.0% decrease relative to baseline sharing intentions for false news in the control condition of those experiments”

In the paper

“The non-evaluation treatments significantly increased sharing discernment (interaction between headline veracity and treatment dummies: b = 0.039, z = 4.974, p < 0.001), which translates into a 70.9% increase over baseline sharing discernment in the control condition of those experiments”

In the paper

“This increase in discernment was again largely driven by the Evaluation treatment significantly decreasing sharing intentions for false news (treatment dummy: b = −0.027, z = −5.548, p < 0.001), which translates into an 8.2% decrease relative to baseline sharing intentions for false news in the control condition”

In the paper

“In total, we meta-analyze k = 20 experiments, with a total of N = 26,863 participants (see Table 2).”

In the paper

“In the more representative samples, across all three measures we find that people who identify as Republican were significantly less discerning in their baseline sharing (in line with what has been observed previously), but partisanship does not significantly moderate the accuracy prompt effect on discernment.”

In the paper

“Interestingly, women are significantly less discerning in the control condition on MTurk, but not in the more representative samples.”

In the paper

“On MTurk, we see the opposite pattern: Republicans were not significantly less discerning in their baseline sharing, yet partisanship does significantly moderate the accuracy prompt on discernment, with the prompts working less well for Republicans”

In the paper

“Critically, the pattern of results for this Twitter data matches what we report above for the survey experiments: Users who are more conservative shared lower quality information at baseline, but we do not find evidence that ideology moderated the effect of the accuracy prompts.”

In the paper

“Importantly, we do find a significant negative relationship between conservatism and baseline sharing discernment (i.e., overall quality of news that is shared) in the control, and this is evident in both sets of subject pools (although the relationship is twice as large in the more representative samples).”

In the paper

“Consistent with our proposed mechanism, the treatment effect is strongly correlated with perceived accuracy in all experiments (Fig. 4): the meta-analytic estimate of the correlation is r = 0.773, z = 19.417, p < 0.001”

In the paper

“Here, we provide evidence, across 20 experiments with U.S. samples, that a variety of accuracy prompts robustly increased social media users’ sharing discernment by decreasing their intentions to share a wide range of false news headlines about politics and COVID-19.”

In the paper

“including this study in our main analysis does not meaningfully alter the results (see SI Section 4)”

In the paper

“Another advantage is that accuracy prompts do not require stakeholders such as technology companies or government regulators to decide what is true versus false.”

In the paper

“Evidence for the role of inattention comes from experiments in which prompting participants to think about the concept of accuracy – for example, by asking them to evaluate the accuracy of a random headline at the beginning of the study – reduces the disconnect between accuracy judgments and sharing intentions, and thereby increases the quality of news shared24–26.”

What the sources say

• Not checked yetcites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source
• Not checked yetcites [25] Epstein, Z. et al. Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online. Harvard Kennedy Sch. Misinformation Rev. https://doi.org/10.37016/mr-2020-71 (2021). source
• Not checked yetcites [26] Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention. Psychol. Sci. 31, 770–780. https://doi.org/10.31234/OSF.IO/UHBK9 (2020). source

In the paper

“Ironically, this occurs despite the fact that an overwhelming majority of participants say that it is important to only share accurate news24.”

What the sources say

• Not checked yetcites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

In the paper

“As a result, study selection was broad and not susceptible to motivated choices about inclusion.”

In the paper

“Furthermore, we included all interventions that could be construed as accuracy prompts – i.e., interventions that occurred prior to the sharing task, that invoked accuracy in some way (such that the concept of accuracy would be primed), and did not provide any specific information about the veracity of any particular headlines (were “content neutral”).”

In the paper

“Instead, they leverage users’ own abilities to identify misinformation and take advantage of the fact that most people want to share accurate content24.”

What the sources say

• Not checked yetcites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source

In the paper

“This approach is superior to simply focusing on the sharing of false news because an intervention that decreases the sharing of both true and false news equally would not indicate that people are focusing more on accuracy40.”

What the sources say

• Not checked yetcites [40] Pennycook, G. & Rand, D. G. Lack of partisan bias in the identification of fake (versus real) news. Trends Cogn. Sci. 25, 725–726 (2021). source

In the paper

“Although not a perfectly controlled comparison, the observation that the combined treatments were roughly twice as effective as Evaluation alone suggests that there are substantial gains from stacking accuracy prompt interventions.”

In the paper

“Importantly, although our meta-analysis was not itself pre-registered, we made the decision about what to include before conducting the meta-analysis.”

In the paper

“There are two main threats to the validity of meta-analytic results: the systematic omission of studies (e.g., publication bias suppressing studies with null results36,37), and the flexible selection of analysis approaches within each study inflating the rate of false positives (e.g., p-hacking38).”

What the sources say

• Not checked yetcites [36] Sutton, A. J., Song, F., Gilbody, S. M. & Abrams, K. R. Modelling publication bias in meta-analysis: A review. Stat. Methods Med. Res. 9, 421–445 (2000). source
• Not checked yetcites [37] Stanley, T. D. Limitations of PET-PEESE and other meta-analysis methods. Soc. Psychol. Personal. Sci. 8, 581–591 (2017). source
• Not checked yetcites [38] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). source

In the paper

“This worse baseline discernment aligns with real-world sharing data where conservatives were found to share more fake news on Twitter10 and Facebook12.”

What the sources say

• Not checked yetcites [10] Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B. & Lazer, D. Fake news on twitter during the 2016 U.S. Presidential election. Science 363, 374–378 (2019). source
• Not checked yetcites [12] Guess, A. M., Nagler, J. & Tucker, J. Less than you think: Prevalence and predictors of fake news dissemination on Facebook. Sci. Adv. 5, eaau4586 (2019). source

In the paper

“Although we have comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference),”

In the paper

“we have comparatively low power for statistically analyzing heterogeneity across studies (16 to 20 studies, depending on the individual difference)”

In the paper

“given the dissociation between accuracy judgments and sharing intentions observed in past work24–26”

What the sources say

• Not checked yetcites [24] Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2 (2021). source
• Not checked yetcites [25] Epstein, Z. et al. Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online. Harvard Kennedy Sch. Misinformation Rev. https://doi.org/10.37016/mr-2020-71 (2021). source
• Not checked yetcites [26] Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention. Psychol. Sci. 31, 770–780. https://doi.org/10.31234/OSF.IO/UHBK9 (2020). source

In the paper

“An important caveat for these analyses is that attention was measured post-treatment, which has the potential to undermine inferences drawn from these data43”

What the sources say

• Not checked yetcites [43] Montgomery, J. M., Nyhan, B. & Torres, M. How conditioning on posttreatment variables can ruin your experiment and what to do about it. Am. J. Pol. Sci. 62, 760–775 (2018). source

In the paper

“Prior work indicates that cognitive reflection is positively associated with truth discernment when making accuracy judgments18,45”

What the sources say

• Not checked yetcites [18] Pennycook, G. & Rand, D. G. The psychology of fake news. Trends Cogn. Sci. 25, 388–402 https://doi.org/10.1016/j.tics.2021.02.007 (2021). source
• Not checked yetcites [45] Pennycook, G. & Rand, D. G. Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition 188, 39–50 (2019). source

In the paper

“Thus, if a group of participants is targeted who have a very difficult time distinguishing between true and false content, or if misinformation is identified that the majority of a particular group of people believe to be true, then accuracy prompts will not improve sharing discernment.”

In the paper

“In most cases, headlines were selected for inclusion based on prior pre-testing; for all experiments using political headlines, the headlines were balanced on partisan lean based on the pre-test results”

In the paper

“Although these inclusion and exclusion criteria were determined prior to running the meta-analysis”

In the paper

“the meta-analysis was not pre-registered and no protocol was prepared”

In the paper

“scholars have argued that replicable priming effects are restricted to “perceptual primes” (e.g., as in semantic priming), which involve (for example) perceiving an ambiguous stimuli in a way that is more consistent with a prime58”

What the sources say

• Not checked yetcites [58] Harris, C., Rohrer, D. & Pashler, H. A train wreck by any other name. Psychological Inq. 32, 17–23 (2021). source

In the paper

“This form of priming is said to occur when “a stimulus influences subsequent behavior without conscious guidance or intention”55”

What the sources say

• Not checked yetcites [55] Sherman, J. W. & Rivers, A. M. There’s nothing social about social priming: derailing the “Train Wreck”. Psychol. Inq. 32, 1–11 (2021). source

In the paper

“Although the benefit of internal meta-analyses is that one can be confident that the entire “file-drawer” is included and that p-hacking has been avoided”

In the paper

“Both authors agreed on the classification of each study as eligible versus ineligible.”

In the paper

“Many studies also prevented respondents who failed trivial attention checks at the study outset from participating, but since no data was collected from these participants, we cannot assess the impact of extreme inattention.”

In the paper

“Several prominent examples of behavioral priming have proven difficult to replicate56,57”

What the sources say

• Not checked yetcites [56] Cesario, J. Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9, 40–48 (2014). source
• Not checked yetcites [57] Molden, D. C. Understanding priming effects in social psychology: An overview and integration. Soc. Cognition 32, 243–249 (2014). source

In the paper

“We use random effects meta-analysis, rather than fixed-effects meta-analysis, because there is reason to expect that the true effect size varies across studies”