The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Scaling up fact-checking using the wisdom of crowds
Open accessAbstract
When rating articles’ accuracy, a small politically balanced crowd of laypeople yields high agreement with fact-checkers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and (ii) it achieves high identification accuracy enabled by the wisdom of crowds (Drolsbach and Pr¨ ollochs 2023a; Allen et al. 2021; Martel et al. 2024)”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Yet, evidence from studies on traditional misinformation shows that crowd-based approaches, where multiple judgments are aggregated, can achieve accuracy levels comparable to those of experts and overcome scalability issues (Allen et al. 2021; Martel et al. 2024; Drolsbach and Pr¨ ollochs 2023a; He et al. 2025; Pilarski and Pr¨ ollochs 2026).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
In the citing paper
“Recent empirical research finds that averaged responses of ordinary people are on par with third-party fact-checkers for both factual claims in articles (47) and overall accuracy of content from URL domain names (48, 49).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Self-citationFake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304
When Corrections Fail: The Persistence of Political Misperceptions
Nyhan, B., & Reifler, J. (2010) · Political Behavior · doi:10.1007/s11109-010-9112-2 · Cited by 3,173 · cited 1×
“Debunking false claims typically reduces incorrect belief ( 5 ) [early concerns about potential “backfire effects” ( 6 , 7 ) have not been supported by subsequent work; for a review, see ( 8 )].”
Misinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920
The Polarizing Impact of Science Literacy and Numeracy on Perceived Climate Change Risks
Braman, D., Kahan, D. M., Peters, E. M., Wittlin, M., Slovic, P., Ouellette, L. L., & Mandel, G. N. (2012) · Scholarly Commons - George Washington University Law School (George Washington University) · Cited by 2,421
The Persuasiveness of Source Credibility: A Critical Review of Five Decades' Evidence
Pornpitakpan, C. (2004) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2004.tb02547.x · Cited by 2,205 · cited 1×
“While a large literature has shown that people preferentially believe information from trusted sources [for a review, see ( 52 )], this work has usually focused on information shared by people who are more or less trusted.”
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022 · cited 4×
“…we cannot rule out this possibility, we did replicate our results by analyzing a previously published dataset of researcher-selected headlines ( 25 ) and found that the crowd performed substantially better than for the articles provided by Facebook (see section S16), suggesting that the…”
InfluentialSelf-citationPennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715 · cited 2×
“…with rapidly evolving facts, such as in the case of the coronavirus disease 2019 (COVID-19) news environment, results for both the crowd and fact-checkers would differ [although studies with researcher-selected true and false COVID-19 headlines find high crowd performance, e.g., ( 39 )].”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- 10.1038/nclimate1547 · DOI · Paper page
Cited by
Sorted by citations
Items 201–235 of 235
The People’s Cure: Sociotechnical Strategies Against Health Misinformation
Jonathan, G. M., Watat, J. K., & Ayeni, F. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14750-9_4
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0006
Creating Ignorance through Absences
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0010
Closing Gaps and Making Connections
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0012
Smithson, M. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.002.0005
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0009
Smithson, M. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.002.0004
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0002
Creating Ignorance through Falsehoods
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0011
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0005
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0007
Smithson, M. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0004
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0008
Ignorance Studies, Where Is Psychology?
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0001
What Do We Think Others Are Missing?
Smithson, M. J. (2026) · Oxford University Press eBooks · doi:10.1093/9780198960614.003.0003
Lavigne, M. (2026) · Journal of Elections Public Opinion and Parties · doi:10.1080/17457289.2026.2669135
Oversight Board and its Legitimacy: Genesis of Self-Regulation from Roman Jurisdiction
Filó, M. D. C. S., & Borges, G. S. (2024) · Pensar - Revista de Ciências Jurídicas · doi:10.5020/2317-2150.2024.16862
Vu, C. P., & Saling, L. L. (2026) · PLoS ONE · doi:10.1371/journal.pone.0348291
“Hence, individuals may reject or ignore debunking, if they assume that fact-checkers are pushing a particular political agenda [13].”
Adena, M., Janowska-Widomska, P., & Harke, J. (2026) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2026.102602
Dahlke, R., Moore, R. C., Adib-Azpeitia, D., Ugander, J. H., & Hancock, J. T. (2026) · Political Communication · doi:10.1080/10584609.2026.2679492
“Wisdom of Crowds (e.g., Allen et al., 2021) or large language model-based approaches may help make progress in this area in the future, but creating and validating such a technique would require a methodology paper on its own.”
Crowdsourced Fact-checking on X: from professional fact-checking to Community Notes
Cantón-Correa, J., & Moreno, M. (2026) · Communication & Society · doi:10.15581/003.39.1.032
Sharma, A., Gupta, D., & Bhatia, V. (2026) · International Conference on Innovations in Computational Intelligence (ICICI) · doi:10.1109/icici68867.2026.11565047
“Professional fact-checking organizations like PolitiFact and Snopes represent the pinnacle of quality in fact-checking, but their centralized and labour-intensive processes render them constitutionally incompatible with the speed at which information spreads today [4].”
MMFN: MLLMs-guided Multi-source Information Fusion Network for Multimodal Fake News Detection
Zhang, R., Ji, S., Lv, J., Li, N., & Li, H. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3820376
Bozkurt, B., Mosleh, M., & Margetts, H. Z. (2026) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-202
Fact-checking in the multipolar AI order: Between epistemic sovereignty and ambivalence
Asmolov, G. (2026) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-201
Chen, X. K., Na, J., & Ye, Y. E. (2026) · Journal of Information Science · doi:10.1177/01655515261467993
Brandtzæg, P. B., Alvestad, S. S., Kunst, J. R., & Følstad, A. (2026) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-206
Fake news detection via wisdom of synthetic and representative crowds
t'Serstevens, F., Cerina, R., & Piccillo, G. (2026) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1093/jrsssa/qnag095 · Cited by 2
Who Chooses to Judge? Endogenous Participation and the Limits of Bridging in CrowdPreprint
Hou, J., & Zhou, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7240218
What Facebook Users Flag as False News: A Mixed-Methods Investigation of User-Reported Links
Holt, A. E. (2026) · Digital Journalism · doi:10.1080/21670811.2026.2703607
Modeling multi-version information propagation with semantic evolution on adaptive temporal networks
Xiao, Y., Liang, Z.-W., Chuan-lin, L., & Wang, Z. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131956
Leveraging Collective Advice-Taking Behavior to Infer Accuracy and Improve the Wisdom of Crowds
Zhang, Y. (2026) · Management Science · doi:10.1287/mnsc.2024.04499
YouCred: An Online Tool Co-Designed with Fact-Checkers for Misinformation Discovery on YouTube
Juneja, P., Xie, D., Ye, G., & Mitra, T. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817042
“…monitoring tools, previous research has explored various methods of determining fact-check-worthy claims in text [41, 55], including crowdsourcing [3, 4, 11, 32, 43, 61, 69], artificial intelligence [13, 30, 107], natural language processing [31, 97], knowledge graphs [88], visual analytics [64],…”
On the Vulnerability of Community Notes to Participation Concentration
Nudo, J., Nemmi, E. N., Loru, E., Mei, A., Quattrociocchi, W., & Cinelli, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38695-3_24
“Therefore, for the analyses explicitly concerned with polarization and selective engagement, which would require the authors’ identifier, we focus on a politically annotated sample of notes constructed by [46] 2 .”
Chen, L., Liu, X., & Fu, L. (2026) · Science Communication · doi:10.1177/10755470261489553
Metadata
- Authors
- Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Nottingham (GB)
- University of Regina (CA)
- Corresponding author
- Allen, Jennifer
- Arechar, Antonio A.
- Rand, David G.
- Venue
- Science Advances
- Published
- 2021-09-03
- Volume
- 7
- Issue
- 36
- Pages
- eabf4393-eabf4393
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC8442902
- OpenAlex
- W3172233798
- Cited by
- 234
- References
- 56
- Citations by year
- 2026: 44 · 2025: 47 · 2024: 48 · 2023: 45 · 2022: 33 · 2021: 15
- FWCI
- 49.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- PsyArXiv (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation (61061)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- fact-checking, misinformation, crowdsourcing, wisdom of crowds, laypeople, political balance, Facebook, news headlines, cognitive reflection, political knowledge, partisanship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
