Can The Crowd Identify Misinformation Objectively?: The Effects of Judgment Scale and Assessor's Background
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“eve evaluations of different articles atfactcheck.orgto assess claims made in American political debates. They then generate labels on a three-level scale: False, Half True, and True. Vosoughi et al. [53] check the consistency between multiple fact-checking websites on three levels: False, Mixed, and True. Tchechmedji…”
BackgroundCognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Analysis of Ordinal Categorical Data
Agresti, A. (2010) · Wiley Series in Probability and Statistics · doi:10.1002/9780470594001 · Cited by 1,620
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
CSI: A Hybrid Deep Model for Fake News Detection
Ruchansky, N., Seo, S., & Liu, Y. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132877 · Cited by 1,039
Fighting misinformation on social media using crowdsourced judgments of news source quality
Pennycook, G., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806781116 · Cited by 947 · cited 1×
“Previous studies have found a positive correlation between cognitive skills (measured by means of the cognitive reflection test or CRT [13]) and the ability to identify true and false news [36–38].”
BackgroundPennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879
Fact Checking: Task definition and dataset construction
Vlachos, A., & Riedel, S. (2014) · Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science · doi:10.3115/v1/w14-2508 · Cited by 402 · cited 1×
“wn labels and rating system used to describe the authenticity of the content. Therefore, to integrate multiple datasets, converging to a common rating scale becomes very important. Vlachos and Riedel [50] aligned labels from Channel 4 and PolitiFact to a five-point scale:False, Mostly False, Half True, Mostly True, an…”
MethodCognitive Biases in Crowdsourcing
Eickhoff, C. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159654 · Cited by 144 · cited 2×
“In the context of crowdsourced relevance judgments, Eickhoff [11] showed how common types of bias can impact the collected judgments and the results of Information Retrieval (IR) evaluation initiatives.”
Cited by
Crowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
The Effects of Crowd Worker Biases in Fact-Checking Tasks
Draws, T., La Barbera, D., Soprano, M., Roitero, K., Ceolin, D., Checco, A., & Mizzaro, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534629 · Cited by 23
“The use of this metric is motivated by earlier work [37, 43] and theoretical reasons [43].”
BackgroundResultInfluentialSelf-citationCombining Human and Machine Confidence in Truthfulness Assessment
Qu, Y., Roitero, K., Barbera, D. L., Spina, D., Mizzaro, S., & Demartini, G. (2023) · Journal of Data and Information Quality · doi:10.1145/3546916 · Cited by 6
Self-citation
Metadata
- Authors
- Roitero, K., Soprano, M., Fan, S., Spina, D., Mizzaro, S., & Demartini, G.
- Venue
- Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2020-07-25
- Pages
- 439-448
- OpenAlex
- W4288080154
- Cited by
- 35
- References
- 13
- FWCI
- 11.52
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can The Crowd Identify Misinformation Objectively?: The Effects of Judgment Scale and Assessor's Background. https://app.cochat.ai/papers/10.1145/3397271.3401112
- The paper
- Roitero, K., Soprano, M., Fan, S., Spina, D., Mizzaro, S., & Demartini, G. (2020). Can The Crowd Identify Misinformation Objectively?: The Effects of Judgment Scale and Assessor's Background. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3397271.3401112
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.