Disentangling item and testing effects in inoculation research on online misinformation: Solomon revisited
Open accessAbstract
Online misinformation is a pervasive global problem. In response, psychologists have recently explored the theory of psychological inoculation: If people are preemptively exposed to a weakened version of a misinformation technique, they can build up cognitive resistance. This study addresses two unanswered methodological questions about a widely adopted online “fake news” inoculation game, Bad News. First, research in this area has often looked at pre- and post-intervention difference scores for the same items, which may imply that any observed effects are specific to the survey items themselves (item effects). Second, it is possible that using a pretest influences the outcome variable of interest, or that the pretest may interact with the intervention (testing effects). We investigate both item and testing effects in two online studies (total N = 2,159) using the Bad News game. For the item effect, we examine if inoculation effects are still observed when different items are used in the pre- and posttest. To examine the testing effect, we use a Solomon’s Three Group Design. We find that inoculation interventions are somewhat influenced by item effects, and not by testing effects. We show that inoculation interventions are effective at improving people’s ability to spot misinformation techniques and that the Bad News game does not make people more skeptical of real news. We discuss the larger relevance of these findings for evaluating real-world psychological interventions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Consumers may take an offensive coping strategy by refuting the claims portrayed in fake content by searching for and presenting contrary evidence to protect other consumers.
“Consumers may even take an offensive coping strategy by refuting the claims portrayed in fake content by searching for and presenting contrary evidence to protect other consumers (Roozenbeek et al., 2021).”
From Deepfakes: Deceptions, mitigations, and opportunities (2023) · cites this paper as Roozenbeek et al., 2021
The Introduction discusses fact-checking, debunking, and inoculation against misinformation, but does not describe consumers searching for and presenting contrary evidence to protect other consumers. Searches for the claimed offensive coping strategy and counterarguing found no passage addressing that behavior. The passages provide no page numbers. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1× · see citing works
“…of misinformation once it has taken hold in memory (Lewandowsky et al., 2012), and the fact that misinformation may spread further, faster, and deeper on social media than other types of news, thus, ensuring that fact-checking efforts are likely to remain behind the curve (Vosoughi et al., 2018).”
BackgroundEvaluating Effect Size in Psychological Research: Sense and Nonsense
Funder, D. C., & Ozer, D. J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847202 · Cited by 3,295 · cited 1× · see citing works
“…effect size differs from set to set, and that the preregistered value of d = 0.30 was too optimistic for Set B. Nonetheless, d = 0.23 (which would have yielded no statistical equivalence to zero) is considered a meaningful effect size for interventions in persuasion research (Funder & Ozer, 2019).”
BackgroundInfluentialMisinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920 · see citing works
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161 · see citing works
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 1× · see citing works
“To investigate whether this equals the absence of any effect of interest (Lakens et al., 2018, 2020), we conducted an equivalence test using two one-sided tests (TOSTs).14 We confirmed statistical equivalence to zero for Set B 2 Set B, t(447) = 2.57, p = .005, with the smallest effect size of…”
MethodInfluentialInoculating the Public against Misinformation about Climate Change
van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017) · Global Challenges · doi:10.1002/gch2.201600008 · Cited by 1,058 · see citing works
Cook, J., Lewandowsky, S., & Ecker, U. K. H. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175799 · Cited by 971 · see citing works
An extension of control group design
Solomon, R. L. (1949) · Psychological Bulletin · doi:10.1037/h0062958 · Cited by 541 · cited 10× · see citing works
“To examine the testing effect, we use a Solomon’s Three Group Design (Solomon, 1949), in which participants are randomly assigned to one of three groups.”
BackgroundMethodInfluentialA Meta-Analysis of Research on Inoculation Theory
Banas, J. A., & Rains, S. A. (2010) · Communication Monographs · doi:10.1080/03637751003758193 · Cited by 510 · see citing works
Pausing to consider why a headline is true or false can help reduce the sharing of false news
Fazio, L. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-009 · Cited by 187 · see citing works
Compton, J. (2012) · The SAGE Handbook of Persuasion: Developments in Theory and Practice · doi:10.4135/9781452218410.n14 · Cited by 158 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- Aldridge V. K. Dovey T. M. Wade A. (2017). Assessing test-retest reliability of psychological measures: Persistent methodological problems. European Psychologist, 22(4), 207- 218. 10.1027/1016-9040/a000298 · DOI
- Basol M. Roozenbeek J. van der Linden S. (2020). Good news about Bad News: Gamified inoculation boosts confidence and cognitive immunity against fake news. Journal of Cognition, 3(1), Article 2. 10.5334/joc.91 PMC6952868 31934684 · DOI · Paper page
- Campbell D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297- 312. http://rogosateaching.com/somgen290/campbell_1957.pdf 13465924
- Compton J. (2018). Inoculation against/with political humor. In Baumgartner J. C. Becker A. B. (Eds.), Political humor in a changing media landscape: A new generation of research (pp. 95- 113). Lexington Books.
- Compton J. (2019). Prophylactic versus therapeutic inoculation treatments for resistance to influence. Communication Theory. Advance online publication. 10.1093/ct/qtz004 · DOI
- Ermert M. (2018). German hate speech legislation receives mixed review at German Internet Governance Forum. Intellectual Property Watch website. https://www.ip-watch.org/2018/11/28/german-hate-speech-legislation-receives-mixed-review-german-internet-governance-forum/
- Guess A. Nagler J. Tucker J. (2019). Less than you think: Prevalence and predictors of fake news dissemination on Facebook. Science Advances, 5(1). 10.1126/sciadv.aau4586 PMC6326755 30662946 · DOI · Paper page
- Lakens D. McLatchie N. Isager P. M. Scheel A. M. Dienes Z. (2020). Improving inferences about null effects with Bayes factors and equivalence tests. Journals of Gerontology: Series B, 75(1), 45- 57. 10.1093/geronb/gby065 29878211 · DOI · Paper page
- Lewandowsky S. Ecker U. K. H. Cook J. (2017). Beyond misinformation: Understanding and coping with the “post-truth” era. Journal of Applied Research in Memory and Cognition, 6(4), 353- 369. 10.1016/j.jarmac.2017.07.008 · DOI · Paper page
- McGuire W. J. (1964). Some contemporary approaches. Advances in Experimental Social Psychology, 1, 191- 229. 10.1016/S0065-2601(08)60052-0 · DOI
- McGuire W. J. Papageorgis D. (1961a). Resistance to persuasion conferred by active and passive prior refutation of the same and alternative counterarguments. Journal of Abnormal and Social Psychology, 63(2), 326- 332. 10.1037/h0048344 · DOI
- McGuire W. J. Papageorgis D. (1961b). The relative efficacy of various types of prior belief-defense in producing immunity against persuasion. Journal of Abnormal and Social Psychology, 62(2), 327- 337. 10.1037/h0042026 13773992 · DOI · Paper page
- Ozbay F. A. Alatas B. (2020). Fake news detection within online social media using supervised artificial intelligence algorithms. Physica A: Statistical Mechanics and Its Applications, 540, Article 123174. 10.1016/j.physa.2019.123174 · DOI
- Pennycook G. Bear A. Collins, Evan T. Rand D. G. (2020). The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings. Management Science. Advance online publication. 10.1287/mnsc.2019.3478 · DOI
- Roozenbeek J. van der Linden S. (2018). The fake news game: Actively inoculating against the risk of misinformation. Journal of Risk Research, 22(5), 570- 580. 10.1080/13669877.2018.1443491 · DOI
- Roozenbeek J. van der Linden S. (2019). Fake news game confers psychological resistance against online misinformation. Palgrave Communications, 5, Article 65. 10.1057/s41599-019-0279-9 · DOI · Paper page
- Roozenbeek J. van der Linden S. Nygren T. (2020, February 3). Prebunking interventions based on “inoculation” theory can reduce susceptibility to misinformation across cultures. Harvard Kennedy School (HKS) Misinformation Review, 1(2). 10.37016//mr-2020-008 · DOI
- Song M. K. Ward S. E. (2015). Assessment effects in educational and psychosocial intervention trials: An important but often-overlooked problem. Research in Nursing & Health, 38(3), 241- 247. 10.1002/nur.21651 25728502 PMC4439294 · DOI
- Tandoc E. C. Jr. Lim Z. W. Ling R. (2018). Defining “Fake News.” Digital Journalism, 6(2), 137- 153. 10.1080/21670811.2017.1360143 · DOI
- van der Linden S. Maibach E. Cook J. Leiserowitz A. Lewandowsky S. (2017). Inoculating against misinformation. Science, 358(6367), 1141- 1142. 10.1126/science.aar4533 29191898 · DOI
- van der Linden S. Roozenbeek J. (2020). Psychological inoculation against fake news. In Greifenader R. Jaffé M. Newman E. Schwarz N. (Eds.), (pp 147-169). The psychology of fake news: Accepting, sharing, and correcting misinformation. Psychology Press.
Cited by
Sorted by citations
Susceptibility to misinformation about COVID-19 around the world
Roozenbeek, J., Schneider, C. R., Dryhurst, S., Kerr, J., Freeman, A. L. J., Recchia, G., van der Bles, A. M., & van der Linden, S. (2020) · Royal Society Open Science · doi:10.1098/rsos.201199 · Cited by 1,447 · see citing works
Self-citationInoculating Against Fake News About COVID-19
van der Linden, S., Roozenbeek, J., & Compton, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.566790 · Cited by 632 · see citing works
Self-citationLong-term effectiveness of inoculation against misinformation: Three longitudinal experiments.
Maertens, R., Roozenbeek, J., Basol, M., & van der Linden, S. (2021) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000315 · Cited by 321 · see citing works
Self-citationPsychological Inoculation against Misinformation: Current Evidence and Future Directions
Traberg, C. S., Roozenbeek, J., & van der Linden, S. (2022) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/00027162221087936 · Cited by 224 · see citing works
Self-citationDeepfakes: Deceptions, mitigations, and opportunities
Mustak, M., Salminen, J., Mäntymäki, M., Rahman, A., & Dwivedi, Y. K. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2022.113368 · Cited by 209 · see citing works
How Can Psychological Science Help Counter the Spread of Fake News?
van der Linden, S., Roozenbeek, J., Maertens, R., Basol, M., Kácha, O., Rathje, S., & Traberg, C. S. (2021) · The Spanish Journal of Psychology · doi:10.1017/sjp.2021.23 · Cited by 71 · see citing works
Self-citationHow to Combat Health Misinformation: A Psychological Approach
Roozenbeek, J., & van der Linden, S. (2022) · American Journal of Health Promotion · doi:10.1177/08901171211070958 · Cited by 60 · see citing works
Self-citation
- Authors
- Roozenbeek, J., Maertens, R., McClanahan, W., & van der Linden, S.
- Affiliations
- University of Cambridge (GB)
- University of Cambridge, Cambridge, UK
- Corresponding author
- Roozenbeek, Jon
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2020-07-16
- Volume
- 81
- Issue
- 2
- Pages
- 340-362
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- PMC
- PMC10621688
- OpenAlex
- W4239239283
- Cited by
- 80
- References
- 11
- Citations by year
- 2026: 9 · 2025: 15 · 2024: 9 · 2023: 25 · 2022: 11 · 2021: 7 · 2020: 4
- FWCI
- 16.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Educational and Psychological Measurement (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by-nc)
- Apollo (University of Cambridge) (submittedVersion)
- Apollo
- Funders
- Economic and Social Research Council (ES/V011960/1, 2268886)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
- Keywords
- psychological inoculation, online misinformation, fake news, testing effect, misinformation detection, pretest effects, intervention evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
