The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Fighting misinformation on social media using crowdsourced judgments of news source quality
Open accessAbstract
Significance: Many people consume news via social media. It is therefore desirable to reduce social media users’ exposure to low-quality news content. One possible intervention is for social media ranking algorithms to show relatively less content from sources that users deem to be untrustworthy. But are laypeople’s judgments reliable indicators of quality, or are they corrupted by either partisan bias or lack of information? Perhaps surprisingly, we find that laypeople—on average—are quite good at distinguishing between lower- and higher-quality sources. These results indicate that incorporating the trust ratings of laypeople into social media ranking algorithms may prove an effective intervention against misinformation, fake news, and news content with heavy political bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent empirical research finds that averaged responses of ordinary people are on par with third-party fact-checkers for both factual claims in articles (47) and overall accuracy of content from URL domain names (48, 49).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [49]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These approaches rely on the collective contributions and crowd signals (Tschiatschek et al. 2018) of a group of people for the aggregation of crowd intelligence to detect fake news (Tchakounté et al. 2020) and to reduce the spread of misinformation on social media (Pennycook and Rand 2019; Micallef et al. 2020).”
From Fake news, disinformation and misinformation in social media: a review (2023) · cites this paper as Pennycook and Rand 2019
The cited source says what the claim attributes to it.
In the citing paper
“Recent research suggests that human reasoning is “meaningfully directed toward the formation of accurate, rather than merely identity-confirming, beliefs” (ref. 24 p. 2525)”
From Factual knowledge can reduce attitude polarization (2025) · cites this paper as [24]
The cited source says what the claim attributes to it.
In the citing paper
“Research on news source quality demonstrates that people in the United States are generally accurate at identifying high and low-quality publishers54”
From Human detection of political speech deepfakes across transcripts, audio, and video (2024) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“The crowdsourced accuracy ratings generated by this process may also be useful for identifying misinformation, as has been found for crowd ratings of source trustworthiness 138,139 .”
From Using social and behavioural science to support COVID-19 pandemic response (2020) · cites this paper as [139]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160 · cited 1×
“This is illustrated by the recent rise of an old form of misinformation: blatantly false news stories that are presented as if they are legitimate (2).”
BackgroundSelf-citationMisinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920
Evidence for a Collective Intelligence Factor in the Performance of Human Groups
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010) · Science · doi:10.1126/science.1193147 · Cited by 2,685 · cited 1×
“Here, we consider an alternative approach that builds off the large literature on collective intelligence and the “ wisdom of crowds ” (10, 11): using crowdsourcing (rather than professional fact-checkers) to assess the reliability of news websites (rather than individual stories), and then…”
BackgroundWhy do humans reason? Arguments for an argumentative theory
Mercier, H., & Sperber, D. (2011) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000968 · Cited by 2,254
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Self-citationFake news on Twitter during the 2016 U.S. presidential election
Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., & Lazer, D. (2019) · Science · doi:10.1126/science.aau2706 · Cited by 1,803 · cited 1×
“(25), as well as websites that generated fake stories (as indicated by snopes.”
BackgroundIdeology, motivated reasoning, and cognitive reflection
Kahan, D. M. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500005271 · Cited by 1,302 · cited 1×
“Specifically, motivated reasoning accounts (17, 30, 31) argue that humans reason akin to lawyers (as opposed to philosophers): We engage analytic thought as a means to justify our prior beliefs and thus facilitate argumentation.”
BackgroundPrior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268
Self-citationCoppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987 · cited 1×
“In the second study, we tested the generalizability of our findings by surveying an additional n = 970 Americans recruited from Lucid, providing a subject pool that is nationally representative on age, gender, ethnicity, and geography (27).”
Result
Research works printed but not linked: 3
- 10.1037/h0101809 · doi:10.1037/h0101809 · Paper page
- G Pennycook DG Rand The implied truth effect: Attaching warnings to a subset of fake news stories increases perceived accuracy of stories without warnings. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3035384. Accessed December 10 2017. (2017).
- RM Faris Partisanship propaganda and disinformation: Online media and the 2016 US Presidential Election. Berkman Klein Center for Internet Society and Research Paper . Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3019414. Accessed August 28 2017. (2017).
Cited by
Using social and behavioural science to support COVID-19 pandemic response
Bavel, J. J. V., Baicker, K., Boggio, P. S., Capraro, V., Cichocka, A., Cikara, M., Crockett, M. J., Crum, A. J., Douglas, K. M., Druckman, J. N., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0884-z · Cited by 4,964
Self-citationThe future of social media in marketing
Appel, G., Grewal, L., Hadi, R., & Stephen, A. T. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00695-1 · Cited by 1,893
“In response, research has begun to explore ways that social media platforms can combat fake news through algorithms that determine the quality of shared content (e.g., Pennycook and Rand 2019).”
BackgroundPennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715
“…2017; Van Bavel & Pereira, 2018), our data are more consistent with recent studies on political misinformation that provide both correlational (Pennycook & Rand, 2019b; including data from Twitter sharing, Mosleh, Pennycook, Arechar, & Rand, 2020) and experimental (Bago, Rand, & Pennycook,…”
BackgroundResultInfluentialSelf-citationThe psychological drivers of misinformation belief and its resistance to correction
Ecker, U. K. H., Lewandowsky, S., Cook, J., Schmid, P., Fazio, L. K., Brashier, N., Kendeou, P., Vraga, E. K., & Amazeen, M. A. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-021-00006-y · Cited by 1,372
Pennycook, G., & Rand, D. G. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.02.007 · Cited by 1,360
“Despite potential concerns about political bias or lack of knowledge, recent work has found high agreement between layperson crowds and fact-checkers when evaluating the trustworthiness of news publishers: the average Democrat, Republican, and fact-checker all gave fake news and hyperpartisan sites very low trust rati…”
Self-citation
Metadata
- Authors
- Pennycook, G., & Rand, D. G.
- Affiliations
- University of Regina (CA)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Pennycook, Gordon
- Rand, David G.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2019-01-28
- Volume
- 116
- Issue
- 7
- Pages
- 2521-2526
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC6377495
- OpenAlex
- W2913081045
- Cited by
- 947
- References
- 26
- Citations by year
- 2026: 99 · 2025: 152 · 2024: 139 · 2023: 126 · 2022: 138 · 2021: 142 · 2020: 125 · 2019: 22
- FWCI
- 214.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Miami Foundation (-1)
- Templeton World Charity Foundation (TWCF0209)
- DOD | Defense Advanced Research Projects Agency (D17AC00005)
- Gouvernement du Canada | Social Sciences and Humanities Research Council of Canada (430-2018-00452)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Sociology and Political Science › Media Influence and Politics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- misinformation, fake news, social media, source trustworthiness, ranking algorithms, partisan bias, political bias
- MeSH terms
- Algorithms, Humans, Judgment, Politics, Social Media, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fighting misinformation on social media using crowdsourced judgments of news source quality. https://app.cochat.ai/papers/10.1073/pnas.1806781116
- The paper
- Pennycook, G., & Rand, D. G. (2019). Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1806781116
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.