Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Fighting misinformation on social media using crowdsourced judgments of news source quality

Open access
Pennycook, Gordon, Rand, David G. · 2019 · Proceedings of the National Academy of Sciences

Abstract

Significance: Many people consume news via social media. It is therefore desirable to reduce social media users’ exposure to low-quality news content. One possible intervention is for social media ranking algorithms to show relatively less content from sources that users deem to be untrustworthy. But are laypeople’s judgments reliable indicators of quality, or are they corrupted by either partisan bias or lack of information? Perhaps surprisingly, we find that laypeople—on average—are quite good at distinguishing between lower- and higher-quality sources. These results indicate that incorporating the trust ratings of laypeople into social media ranking algorithms may prove an effective intervention against misinformation, fake news, and news content with heavy political bias.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1806781116

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Average trust ratings for each source among Democrats (x axis) and Republicans (y axis), in study 1 run on MTurk (A) and study 2 run on Lucid (B). Sources that are trusted equally by Democratic and Republican participants would fall along the solid line down the middle of the figure; sources trusted more by Democratic participants would fall below the line; and sources trusted more by Republican participants would fall above the line. Source names are shown for outlets with 33% familiarity or higher in study 1 and 25% familiarity or higher in study 2, when equally weighting Democratic and Republican participants. Dems, Democrats; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.
Fig. 1. Average trust ratings for each source among Democrats (x axis) and Republicans (y axis), in study 1 run on MTurk (A) and study 2 run on Lucid (B). Sources that are trusted equally by Democratic and Republican participants would fall along the solid line down the middle of the figure; sources trusted more by Democratic participants would fall below the line; and sources trusted more by Republican participants would fall above the line. Source names are shown for outlets with 33% familiarity or higher in study 1 and 25% familiarity or higher in study 2, when equally weighting Democratic and Republican participants. Dems, Democrats; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.CC BY-NC-ND · PubMed Central
Fig. 2. Average trust ratings given by professional fact-checkers (n = 8) for each source in study 2.
Fig. 2. Average trust ratings given by professional fact-checkers (n = 8) for each source in study 2.CC BY-NC-ND · PubMed Central
Fig. 3. Average trust ratings for each source among professional fact-checkers and Democratic (A) or Republican (B) participants in study 2, run on Lucid. Dems, Democrats; FCers, professional fact-checkers; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.
Fig. 3. Average trust ratings for each source among professional fact-checkers and Democratic (A) or Republican (B) participants in study 2, run on Lucid. Dems, Democrats; FCers, professional fact-checkers; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 26 references

Research works printed but not linked: 3

  1. 10.1037/h0101809 · doi:10.1037/h0101809 · Paper page
  2. G Pennycook DG Rand The implied truth effect: Attaching warnings to a subset of fake news stories increases perceived accuracy of stories without warnings. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3035384. Accessed December 10 2017. (2017).
  3. RM Faris Partisanship propaganda and disinformation: Online media and the 2016 US Presidential Election. Berkman Klein Center for Internet Society and Research Paper . Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3019414. Accessed August 28 2017. (2017).

Cited by

See all 947

Metadata

Authors
Pennycook, G., & Rand, D. G.
Affiliations
  • University of Regina (CA)
  • Massachusetts Institute of Technology (US)
Corresponding author
  • Pennycook, Gordon
  • Rand, David G.
Venue
Proceedings of the National Academy of Sciences
Published
2019-01-28
Volume
116
Issue
7
Pages
2521-2526
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W2913081045
Cited by
947
References
26
Citations by year
2026: 99 · 2025: 152 · 2024: 139 · 2023: 126 · 2022: 138 · 2021: 142 · 2020: 125 · 2019: 22
FWCI
214.50
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
  • Miami Foundation (-1)
  • Templeton World Charity Foundation (TWCF0209)
  • DOD | Defense Advanced Research Projects Agency (D17AC00005)
  • Gouvernement du Canada | Social Sciences and Humanities Research Council of Canada (430-2018-00452)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Sociology and Political Science › Media Influence and Politics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
misinformation, fake news, social media, source trustworthiness, ranking algorithms, partisan bias, political bias
MeSH terms
Algorithms, Humans, Judgment, Politics, Social Media, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Fighting misinformation on social media using crowdsourced judgments of news source quality. https://app.cochat.ai/papers/10.1073/pnas.1806781116
The paper
Pennycook, G., & Rand, D. G. (2019). Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1806781116

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.