Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dual-Processing Accounts of Reasoning, Judgment, and Social Cognition
Evans, J. S. B. T. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093629 · Cited by 5,137
The Elaboration Likelihood Model of Persuasion
Petty, R. E., & Cacioppo, J. T. (1986) · Communication and Persuasion · doi:10.1007/978-1-4612-4964-1_1 · Cited by 3,202
Why Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706
Prior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268
Fighting misinformation on social media using crowdsourced judgments of news source quality
Pennycook, G., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806781116 · Cited by 947
Scepticism and uncertainty about climate change: Dimensions, determinants and change over time
Whitmarsh, L. (2011) · Global Environmental Change · doi:10.1016/j.gloenvcha.2011.01.016 · Cited by 890
Pennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879
Science audiences, misinformation, and fake news
Scheufele, D. A., & Krause, N. M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1805871115 · Cited by 878
Metzger, M. J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20672 · Cited by 871
Political Extremism Predicts Belief in Conspiracy Theories
van Prooijen, J.-W., Krouwel, A. P. M., & Pollet, T. V. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550614567356 · Cited by 626 · cited 1×
“with traction across the political spectrum as are a number of conspiracy theories, with the former attitude now represented in general crowds in contrast to the continued fringe nature of the latter [34, 70]. The relationship between credibility and a strong partisan perspective may not exist in these cases, but we m…”
Cited by
Fake news, disinformation and misinformation in social media: a review
Aïmeur, E., Amri, S., & Brassard, G. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01028-5 · Cited by 772
Scaling up fact-checking using the wisdom of crowds
Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G. (2021) · Science Advances · doi:10.1126/sciadv.abf4393 · Cited by 234
“…demonstrated that laypeople can discern truth from falsehoods on stimulus sets curated by experimenters ( 25 , 27 ); for example, Bhuiyan et al. ( 28 ) found that laypeople ratings were highly correlated with expert ratings for claims about scientific topics that had a “high degree of consensus…”
Jahanbakhsh, F., Zhang, A. X., Berinsky, A. J., Pennycook, G., Rand, D. G., & Karger, D. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449092 · Cited by 112
“Some of these initiatives include detecting false or misleading information using machine learning algorithms [12, 57, 67] and crowdsourcing [5, 9, 19, 33, 54], identifying bad actors and helping good actors differentiate themselves1 [82], and providing fact-checked information related to circulated news claims [23, 3…”
BackgroundSelf-citationThe Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., & Bernstein, M. S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445423 · Cited by 101
Crowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
Metadata
- Authors
- Bhuiyan, M. M., Zhang, A. X., Sehat, C. M., & Mitra, T.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-10-14
- Volume
- 4
- Issue
- CSCW2
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3080732897
- Cited by
- 67
- References
- 16
- FWCI
- 11.24
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria. https://app.cochat.ai/papers/10.1145/3415164
- The paper
- Bhuiyan, M. M., Zhang, A. X., Sehat, C. M., & Mitra, T. (2020). Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3415164
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.