Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes
The DOI matches its Crossref record; 34 of the 38 references it cites were found in a registry or an index.
Abstract
Wikipedia is a major source of information for many people. However, false information on Wikipedia raises concerns about its credibility. One way in which false information may be presented on Wikipedia is in the form of hoax articles, i.e., articles containing fabricated facts about nonexistent entities or events. In this paper we study false information on Wikipedia by focusing on the hoax articles that have been created throughout its history. We make several contributions. First, we assess the real-world impact of hoax articles by measuring how long they survive before being debunked, how many pageviews they receive, and how heavily they are referred to by documents on the Web. We find that, while most hoaxes are detected quickly and have little impact on Wikipedia, a small number of hoaxes survive long and are well cited across the Web. Second, we characterize the nature of successful hoaxes by comparing them to legitimate articles and to failed hoaxes that were discovered shortly after being created. We find characteristic differences in terms of article structure and content, embeddedness into the rest of Wikipedia, and features of the editor who created the hoax. Third, we successfully apply our findings to address a series of classification tasks, most notably to determine whether a given article is a hoax. And finally, we describe and evaluate a task involving humans distinguishing hoaxes from non-hoaxes. We find that humans are not particularly good at the task and that our automated classifier outperforms them by a big margin.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, web-scale corpora are known to contain inaccuracies, inconsistencies, and biases (Kumar et al., 2016;Greenstein and Zhu, 2012).”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Kumar et al., 2016
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Collective dynamics of ‘small-world’ networks
Watts, D. J., & Strogatz, S. H. (1998) · Nature · doi:10.1038/30918 · Cited by 43,060
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Information credibility on twitter
Castillo, C., Mendoza, M., & Poblete, B. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963500 · Cited by 2,593
The spreading of misinformation online
Del Vicario, M., Bessi, A., Zollo, F., Petroni, F., Scala, A., Caldarelli, G., Stanley, H. E., & Quattrociocchi, W. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517441113 · Cited by 2,407 · cited 4×
“, in Facebook [9, 15], Twitter [16], and Sina Weibo [42].”
BackgroundInfluentialBelieve it or not: Factors influencing credibility on the Web
Wathen, C. N., & Burkell, J. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10016 · Cited by 1,084
Prominent Features of Rumor Propagation in Online Social Media
Kwon, S., Cha, M., Jung, K., Chen, W., & Wang, Y. (2013) · 2013 IEEE 13th International Conference on Data Mining · doi:10.1109/icdm.2013.61 · Cited by 859 · cited 1×
“[21] identify temporal, structural, and linguistic features of rumors on Twitter; Gupta et al.”
BackgroundLimiting the spread of misinformation in social networks
Budak, C., Agrawal, D., & El Abbadi, A. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963499 · Cited by 807
(2011) · Cited by 724
What makes Web sites credible?: a report on a large quantitative study
Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., et al. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365037 · Cited by 685 · cited 2×
“Such survey-based studies have been carried out both on the Web in general [12, 13, 26] as well as on Twitter in particular [28].”
BackgroundAutomatic detection of rumor on Sina Weibo
Yang, F., Liu, Y., Yu, X., & Yang, M. (2012) · Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics · doi:10.1145/2350190.2350203 · Cited by 680 · cited 2×
“More related to our present work is prior research that aims to build automatic methods for assessing the credibility of a given set of social media posts [6, 22, 42, 43].”
BackgroundGupta, A., Lamba, H., Kumaraguru, P., & Joshi, A. (2013) · Proceedings of the 22nd International Conference on World Wide Web Companion · doi:10.1145/2487788.2488033 · Cited by 573 · cited 7×
“There is a rich line of empirical investigations and case-based studies of how this propagation happens, e.g. , in Facebook [9, 15], Twitter [16], and Sina Weibo [42].”
BackgroundInfluentialHow do users evaluate the credibility of Web sites?: a study with over 2,500 participants
Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003) · Proceedings of the 2003 conference on Designing for user experiences · doi:10.1145/997078.997097 · Cited by 565
Spread of (mis)information in social networks
Acemoglu, D., Ozdaglar, A., & ParandehGheibi, A. (2010) · Games and Economic Behavior · doi:10.1016/j.geb.2010.01.005 · Cited by 444 · cited 1×
“Additionally, researchers have proposed theoretical models of how rumors and misinformation propagate in social networks and how their spread may be contained [32, 1].”
BackgroundA content-driven reputation system for the wikipedia
Adler, B. T., & de Alfaro, L. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242608 · Cited by 440
Tweeting is believing?: understanding microblog credibility perceptions
Morris, M. R., Counts, S., Roseway, A., Hoff, A., & Schwarz, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145274 · Cited by 411
Friggeri, A., Adamic, L., Eckles, D., & Cheng, J. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14559 · Cited by 382 · cited 4×
“, in Facebook [9, 15], Twitter [16], and Sina Weibo [42].”
BackgroundInfluentialSize matters: word count as a measure of quality on wikipedia
Blumenstock, J. E. (2008) · Proceedings of the 17th international conference on World Wide Web · doi:10.1145/1367497.1367673 · Cited by 270
Collective attention in the age of (mis)information
Mocanu, D., Rossi, L., Zhang, Q., Karsai, M., & Quattrociocchi, W. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.01.024 · Cited by 242
Measuring article quality in wikipedia: models and evaluation
Hu, M., Lim, E.-P., Sun, A., Lauw, H. W., & Vuong, B.-Q. (2007) · Proceedings of the sixteenth ACM conference on Conference on information and knowledge management · doi:10.1145/1321440.1321476 · Cited by 242
Containment of misinformation spread in online social networks
Nguyen, N. P., Yan, G., Thai, M. T., & Eidenbenz, S. (2012) · Proceedings of the 4th Annual ACM Web Science Conference · doi:10.1145/2380718.2380746 · Cited by 239
Fact-checking Effect on Viral Hoaxes: A Model of Misinformation Spread in Social Networks
Tambuscio, M., Ruffo, G., Flammini, A., & Menczer, F. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742572 · Cited by 158
Disinformation and misinformation through the internet: Findings of an exploratory study
Hernon, P. (1995) · Government Information Quarterly · doi:10.1016/0740-624x(95)90052-7 · Cited by 153
Computing trust from revision history
Zeng, H., Alhossaini, M. A., Ding, L., Fikes, R. E., & McGuinness, D. L. (2006) · Conference on Privacy, Security and Trust · doi:10.1145/1501434.1501445 · Cited by 134
Assessing the quality of Wikipedia articles with lifecycle based metrics
Wöhner, T., & Peters, R. (2009) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/1641309.1641333 · Cited by 112
Using Deep Linguistic Features for Finding Deceptive Opinion Spam
Xu, Q., & Zhao, H. (2012) · Cited by 82
On measuring the quality of Wikipedia articles
la Calzada, G. D., & Dekhtyar, A. (2010) · doi:10.1145/1772938.1772943 · Cited by 77
Why Do Social Media Users Share Misinformation?
Chen, X., Sin, S.-C. J., Theng, Y.-L., & Lee, C. S. (2015) · Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries · doi:10.1145/2756406.2756941 · Cited by 70 · cited 1×
“But there is also a dark side to the un-reigned proliferation of information over the Web: it has become a breeding ground for false information [6, 7, 12, 15, 19, 43].”
BackgroundMeasuring Qualities of Articles Contributed by Online Communities
Lim, E., Vuong, B.-Q., Lauw, H. W., & Sun, A. (2006) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2006.115 · Cited by 51
How Credible is Information on the Web: Reflections on Misinformation and Disinformation
Keshavarz, H. (2014) · E-LIS Repository (University of Naples Federico II) · Cited by 41
(2008) · Cited by 38
A Functional Analysis of Disinformation
(2014) · iConference 2014 Proceedings · doi:10.9776/14278 · Cited by 34
Computing Trust from Revision History
Zeng, H., Alhossaini, M. A., Li, D., Fikes, R. E., & McGuinness, D. L. (2006) · doi:10.21236/ada454704 · Cited by 21
Atheism and the case against Christ
(2013) · Choice Reviews Online · doi:10.5860/choice.50-4948 · Cited by 4
Research works printed but not linked: 5
- R. DeNardo. The Captain's Propensity: The Andromeda Incident II. Strategic Book Publishing, 2013.
- H. Frankfurt. On bullshit. Raritan Quarterly Review, 6(2):81--100, 1986.
- 10.1002/asi.v58:13 · doi:10.1002/asi.v58:13
- K. Morris. After a half-decade, massive Wikipedia hoax finally exposed. Website, 2013. http://www.dailydot.com/news/wikipedia-bicholim-conflict-hoax-deleted (accessed Oct. 16, 2015).
- 10.5555/2145432.2145602 · doi:10.5555/2145432.2145602
Cited by
Shu, K., Mahudeswaran, D., Wang, S., Lee, D., & Liu, H. (2020) · Big Data · doi:10.1089/big.2020.0062 · Cited by 1,151
“For example, some individuals and organizations spread fake news in social media for financial and political gains.(2,3) It is also reported that fake news has an influence on the 2016 U.”
BackgroundCSI: A Hybrid Deep Model for Fake News Detection
Ruchansky, N., Seo, S., & Liu, Y. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132877 · Cited by 1,039
“From a slightly dierent point of view, one popular approach has been to measure the response an article received by studying its propagation on a social graph [12, 16, 17, 35].”
BackgroundAn overview of online fake news: Characterization, detection, and discussion
Zhang, X., & Ghorbani, A. A. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.03.004 · Cited by 997
Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking
Rashkin, H., Choi, E., Jang, J. Y., Volkova, S., & Choi, Y. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1317 · Cited by 870
“Biyani et al. (2016) examine the unique linguistic styles found in clickbait articles, and Kumar et al. (2016) also characterize hoax documents on Wikipedia.”
BackgroundInfluence of fake news in Twitter during the 2016 US presidential election
Bovet, A., & Makse, H. A. (2019) · CUNY Academic Works (City University of New York) · Cited by 882
Metadata
- Authors
- Kumar, S., West, R., & Leskovec, J.
- Affiliations
- University of Maryland, College Park (US)
- Stanford University (US)
- Venue
- Proceedings of the 25th International Conference on World Wide Web
- Published
- 2016-04-11
- Pages
- 591-602
- OpenAlex
- W2340254858
- Cited by
- 310
- References
- 38
- Citations by year
- 2026: 3 · 2025: 17 · 2024: 27 · 2023: 28 · 2022: 29 · 2021: 71 · 2020: 35 · 2019: 39 · 2018: 34 · 2017: 23 · 2016: 4
- FWCI
- 105.59
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International World Wide Web Conference (WWW) (publishedVersion)
- Funders
- Stanford Data Science Initiative
- Defense Advanced Research Projects Agency (XDATA SIMPLEX)
- SAP
- Volkswagen
- Boeing
- Army Research Office (MURI W911NF11103 W911NF1410358 W911NF09102)
- Yahoo
- Wikimedia Research Fellowship
- National Science Foundation (CNS-1010921 IIS-1149837)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Communication › Wikis in Education and Collaboration
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- Wikipedia, disinformation, misinformation, hoax detection, page views, web citations, article structure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.