CoChat
Sign Up
Cited by 235Misinformation and Its ImpactsMobile Crowdsensing and CrowdsourcingSpam and Phishing Detection

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.

Scaling up fact-checking using the wisdom of crowds

Open access
Allen, Jennifer, Arechar, Antonio A., Pennycook, Gordon, Rand, David G.·2021·Science Advances
DOI 10.1126/sciadv.abf4393PMC PMC8442902
Open this paper in CoChat — free accountOpen PDF

Abstract

When rating articles’ accuracy, a small politically balanced crowd of laypeople yields high agreement with fact-checkers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1126/sciadv.abf4393

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Correlation across articles between (i) politically balanced layperson aggregate accuracy ratings based on reading the headline and lede and (ii) average fact-checker research-based aggregate accuracy ratings, as a function of the number of layperson ratings per article. Laypeople are grouped by condition (Source versus No Source). For results showing up to 100 laypeople per article, see fig. S7; for results using the average of the correlations with a single fact-checker rather than the correlation with the average of the fact-checker ratings, see fig. S9. Panels show results for (A) all articles, (B) nonpolitical articles, and (C) political articles. The dashed line indicates the average Pearson correlation between fact-checkers (all articles, r = 0.62; nonpolitical articles, r = 0.69; political articles, r = 0.56). Error bars indicate 95% confidence intervals.
Fig. 1. Correlation across articles between (i) politically balanced layperson aggregate accuracy ratings based on reading the headline and lede and (ii) average fact-checker research-based aggregate accuracy ratings, as a function of the number of layperson ratings per article. Laypeople are grouped by condition (Source versus No Source). For results showing up to 100 laypeople per article, see fig. S7; for results using the average of the correlations with a single fact-checker rather than the correlation with the average of the fact-checker ratings, see fig. S9. Panels show results for (A) all articles, (B) nonpolitical articles, and (C) political articles. The dashed line indicates the average Pearson correlation between fact-checkers (all articles, r = 0.62; nonpolitical articles, r = 0.69; political articles, r = 0.56). Error bars indicate 95% confidence intervals.CC BY · PubMed Central
Fig. 2. Classifying articles as true versus non-true based on layperson aggregate Likert ratings. (A and B) AUC scores as a function of the number of layperson ratings per article and source condition. AUC is calculated using a model in which the average layperson aggregate Likert rating is used to predict the modal fact-checker categorical rating, where the fact-checker rating is coded as “1” if the modal rating is “True” and “0” otherwise. Error bars indicate 95% confidence intervals. For full receiver operating characteristics curves using a politically balanced crowd of size 26, see section S14. (C) Out-of-sample accuracy for ratings from a politically-balanced crowd of size 26 given source information, calculated separately for unanimous and non-unanimous headlines, as the proportion of unanimous headlines in the sample increases. (D) Cutoff for classifying an article as “True” as the proportion of unanimous headlines in the sample increases.
Fig. 2. Classifying articles as true versus non-true based on layperson aggregate Likert ratings. (A and B) AUC scores as a function of the number of layperson ratings per article and source condition. AUC is calculated using a model in which the average layperson aggregate Likert rating is used to predict the modal fact-checker categorical rating, where the fact-checker rating is coded as “1” if the modal rating is “True” and “0” otherwise. Error bars indicate 95% confidence intervals. For full receiver operating characteristics curves using a politically balanced crowd of size 26, see section S14. (C) Out-of-sample accuracy for ratings from a politically-balanced crowd of size 26 given source information, calculated separately for unanimous and non-unanimous headlines, as the proportion of unanimous headlines in the sample increases. (D) Cutoff for classifying an article as “True” as the proportion of unanimous headlines in the sample increases.CC BY · PubMed Central
Fig. 3. Comparing crowds with different layperson compositions to a baseline, politically balanced crowd. (A) Pearson correlations between the average aggregate accuracy rating of a crowd of size 26 and the average aggregate accuracy rating of the fact-checkers. (B) AUC for the average aggregate accuracy rating of a crowd of size 26 predicting whether the modal fact-checker categorical rating is true. For both (A) and (B), we compare the baseline to a crowd of only Democrats versus only Republicans, a politically balanced crowd of participants who scored above the median on the CRT versus at or below the median on the CRT, and a politically balanced crowd of participants who score above the median on political knowledge versus at or below the median on political knowledge. Means and CIs are generated using bootstraps with 1000 iterations (see section S2 for details). For analysis comparing political to nonpolitical headlines, see section S13.
Fig. 3. Comparing crowds with different layperson compositions to a baseline, politically balanced crowd. (A) Pearson correlations between the average aggregate accuracy rating of a crowd of size 26 and the average aggregate accuracy rating of the fact-checkers. (B) AUC for the average aggregate accuracy rating of a crowd of size 26 predicting whether the modal fact-checker categorical rating is true. For both (A) and (B), we compare the baseline to a crowd of only Democrats versus only Republicans, a politically balanced crowd of participants who scored above the median on the CRT versus at or below the median on the CRT, and a politically balanced crowd of participants who score above the median on political knowledge versus at or below the median on political knowledge. Means and CIs are generated using bootstraps with 1000 iterations (see section S2 for details). For analysis comparing political to nonpolitical headlines, see section S13.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 56 references

Research works printed but not linked: 1

  1. 10.1038/nclimate1547 · DOI · Paper page

Cited by

Sorted by citations

See all 235

Related

Metadata

Authors
Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G.
Affiliations
  • Massachusetts Institute of Technology (US)
  • University of Nottingham (GB)
  • University of Regina (CA)
Corresponding author
  • Allen, Jennifer
  • Arechar, Antonio A.
  • Rand, David G.
ORCID
Venue
Science Advances
Publisher
American Association for the Advancement of Science (AAAS)
Published
2021-09-03
Volume
7
Issue
36
Pages
eabf4393-eabf4393
ISSN
2375-2548 (ISSN-L)
DOI
10.1126/sciadv.abf4393
PMC
PMC8442902
OpenAlex
W3172233798
Cited by
234
References
56
Citations by year
2026: 44 · 2025: 47 · 2024: 48 · 2023: 45 · 2022: 33 · 2021: 15
FWCI
49.20
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Science Advances (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Funders
John Templeton Foundation (61061)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
fact-checking, misinformation, crowdsourcing, wisdom of crowds, laypeople, political balance, Facebook, news headlines, cognitive reflection, political knowledge, partisanship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review