Automatically Appraising the Credibility of Vaccine-Related Web Pages Shared on Social Media: A Twitter Surveillance Study
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Background: Tools used to appraise the credibility of health information are time-consuming to apply and require context-specific expertise, limiting their use for quickly identifying and mitigating the spread of misinformation as it emerges. Objective: The aim of this study was to estimate the proportion of vaccine-related Twitter posts linked to Web pages of low credibility and measure the potential reach of those posts. Methods: Sampling from 143,003 unique vaccine-related Web pages shared on Twitter between January 2017 and March 2018, we used a 7-point checklist adapted from validated tools and guidelines to manually appraise the credibility of 474 Web pages. These were used to train several classifiers (random forests, support vector machines, and recurrent neural networks) using the text from a Web page to predict whether the information satisfies each of the 7 criteria. Estimating the credibility of all other Web pages, we used the follower network to estimate potential exposures relative to a credibility score defined by the 7-point checklist. Results: The best-performing classifiers were able to distinguish between low, medium, and high credibility with an accuracy of 78% and labeled low-credibility Web pages with a precision of over 96%. Across the set of unique Web pages, 11.86% (16,961 of 143,003) were estimated as low credibility and they generated 9.34% (1.64 billion of 17.6 billion) of potential exposures. The 100 most popular links to low credibility Web pages were each potentially seen by an estimated 2 million to 80 million Twitter users globally. Conclusions: The results indicate that although a small minority of low-credibility Web pages reach a large audience, low-credibility Web pages tend to reach fewer users than other Web pages overall and are more commonly shared within certain subpopulations. An automatic credibility appraisal tool may be useful for finding communities of users at higher risk of exposure to low-credibility vaccine communications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 2×
“Our study is also not directly comparable with previous studies that have examined how misinformation spreads through social media [2-6].”
BackgroundResultLazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“An example is the Universal Language Model Fine-Tuning method [50], which was proposed and evaluated on natural language processing tasks.”
MethodMisinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920
The spreading of misinformation online
Del Vicario, M., Bessi, A., Zollo, F., Petroni, F., Scala, A., Caldarelli, G., Stanley, H. E., & Quattrociocchi, W. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517441113 · Cited by 2,407
Fake news on Twitter during the 2016 U.S. presidential election
Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., & Lazer, D. (2019) · Science · doi:10.1126/science.aau2706 · Cited by 1,803
Healthcare information on YouTube: A systematic review
Madathil, K. C., Rivera-Rodriguez, A. J., Greenstein, J. S., & Gramopadhye, A. K. (2015) · Health Informatics Journal · doi:10.1177/1460458213512220 · Cited by 1,290 · cited 1×
“It is likely that a substantial proportion of Instagram, Facebook, and YouTube Web pages would receive a low-credibility score if they were assessed [56-58], which means we may have underestimated the sharing of low-credibility vaccine-related communications on Twitter.”
BackgroundPrior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268 · cited 1×
“Methods for inoculating against misinformation by providing warnings immediately before access have mixed results [10,54,55].”
BackgroundWeaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149
CSI: A Hybrid Deep Model for Fake News Detection
Ruchansky, N., Seo, S., & Liu, Y. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132877 · Cited by 1,039
Cited by
What social media told us in the time of COVID-19: A scoping review
Tsao, S.-F., Chen, H., Tisseverasinghe, T., Yang, Y., Li, L., & Butt, Z. A. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30315-0 · Cited by 816
Top concerns of tweeters during the COVID-19 pandemic: Infoveillance study
Abd-Alrazaq, A., Alhuwail, D., Househ, M., Hamdi, M., & Shah, Z. (2020) · Journal of Medical Internet Research · doi:10.2196/19016 · Cited by 774
“Other studies analyzed twitter data for sentiment analysis [12] and the use of Twitter to propagate credible vaccine-related web pages [8].”
BackgroundSelf-citationSocial media and health care, part I: literature review of social media use by health care providers
Farsi, D. (2021) · Journal of Medical Internet Research · doi:10.2196/23205 · Cited by 299
Metadata
- Authors
- Shah, Z., Surian, D., Dyda, A., Coiera, E., Mandl, K. D., & Dunn, A. G.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-11-04
- Volume
- 21
- Issue
- 11
- Pages
- e14007
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2971934325
- Cited by
- 58
- References
- 18
- FWCI
- 17.17
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.