CoChat
Sign Up
Cited by 84Misinformation and Its ImpactsMobile Crowdsensing and CrowdsourcingEpistemology, Ethics, and Metaphysics

Crowds Can Effectively Identify Misinformation at Scale

Open access
Martel, Cameron, Allen, Jennifer, Pennycook, Gordon, Rand, David G.·2024·Perspectives on Psychological Science
DOI 10.1177/17456916231190388
Open this paper in CoChat — free accountOpen PDF

Abstract

Identifying successful approaches for reducing the belief and spread of online misinformation is of great importance. Social media companies currently rely largely on professional fact-checking as their primary mechanism for identifying falsehoods. However, professional fact-checking has notable limitations regarding coverage and speed. In this article, we summarize research suggesting that the "wisdom of crowds" can be harnessed successfully to help identify misinformation at scale. Despite potential concerns about the abilities of laypeople to assess information quality, recent evidence demonstrates that aggregating judgments of groups of laypeople, or crowds, can effectively identify low-quality news sources and inaccurate news posts: Crowd ratings are strongly correlated with fact-checker ratings across a variety of studies using different designs, stimulus sets, and subject pools. We connect these experimental findings with recent attempts to deploy crowdsourced fact-checking in the field, and we close with recommendations and future directions for translating crowdsourced ratings into effective interventions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/17456916231190388

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Crowd-based approaches can achieve accuracy levels comparable to experts and overcome scalability issues.

    “Yet, evidence from studies on traditional misinformation shows that crowd-based approaches, where multiple judgments are aggregated, can achieve accuracy levels comparable to those of experts and overcome scalability issues (Allen et al. 2021; Martel et al. 2024; Drolsbach and Pr¨ ollochs 2023a; He et al. 2025; Pilarski and Pr¨ ollochs 2026).”

    From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Martel et al. 2024

  • Community Notes achieves high identification accuracy enabled by the wisdom of crowds.

    “and (ii) it achieves high identification accuracy enabled by the wisdom of crowds (Drolsbach and Pr¨ ollochs 2023a; Allen et al. 2021; Martel et al. 2024)”

    From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Martel et al. 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Adamic, L. A., Zhang, J., Bakshy, E., & Ackerman, M. S. (2008, April). Knowledge sharing and yahoo answers: Everyone knows something. In Proceedings of the 17th International Conference on World Wide Web (pp. 665- 674). Association for Computing Machinery. https://doi .org/10.1145/1367497.1367587 · DOI
  2. Adler, B. T., & De Alfaro, L. (2007, May). A content-driven reputation system for the Wikipedia. In Proceedings of the 16th International Conference on World Wide Web (pp. 261-270). Association for Computing Machinery. https://doi.org/10.1145/1242572.1242608 · DOI
  3. Allen, J., Martel, C., & Rand, D. G. (2022, April). Birds of a feather don't fact-check each other: Partisanship and the evaluation of news in Twitter's Birdwatch crowd- sourced fact-checking program. In CHI Conference on Human Factors in Computing Systems (pp. 1-19). Association for Computing Machinery. https://doi.org/10 .1145/3491102.3502040 · DOI
  4. Almaatouq, A., Noriega-Campero, A., Alotaibi, A., Krafft, P. M., Moussaid, M., & Pentland, A. (2020). Adaptive social net- works promote the wisdom of crowds. Proceedings of the National Academy of Sciences, USA, 117(21), 11379-11386.
  5. Amazeen, M. A. (2016). Checking the fact-checkers in 2008: Predicting political ad scrutiny and assessing consistency. Journal of Political Marketing, 15(4), 433-464.
  6. Bahrami, B., Olsen, K., Latham, P. E., Roepstorff, A., Rees, G., & Frith, C. D. (2010). Optimally interacting minds. Science, 329(5995), 1081-1085.
  7. Bhuiyan, M. M., Zhang, A. X., Sehat, C. M., & Mitra, T. (2020). Investigating differences in crowdsourced news credibility assessment: Raters, tasks, and expert criteria. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1-26.
  8. Bode, L., & Vraga, E. (2021). The Swiss cheese model for mitigating online misinformation. Bulletin of the Atomic Scientists, 77(3), 129-133.
  9. Celadin, T., Capraro, V., Pennycook, G., & Rand, D. G. (2023). Displaying news source trustworthiness ratings reduces sharing intentions for false news posts. Journal of Online Trust and Safety, 1(5). https://doi.org/10.54501/jots.v1i5.100 · DOI
  10. Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S., Kim, J., & Lasecki, W. S. (2019). Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1-25.
  11. Coscia, M., & Rossi, L. (2020). Distortions of political bias in crowdsourced misinformation flagging. Journal of the Royal Society Interface, 17(167), Article 20200020.
  12. Dai, W. D., Jin, G., Lee, J., & Luca, M. (2018). Aggregation of consumer ratings: An application to yelp.com. Quantitative Marketing and Economics, 16(3), 289-339.
  13. Duncan, M. (2022). Selective rating: Partisan bias in crowd- sourced news rating systems. Journal of Information Technology & Politics, 19(3), 360-375.
  14. Flanigan, B., Gölz, P., Gupta, A., Hennig, B., & Procaccia, A. D. (2021). Fair algorithms for selecting citizens' assem- blies. Nature, 596(7873), 548-552.
  15. Fradkin, A., & Holtz, D. (2021). More reviews may not help: Evidence from incentivized first reviews on Airbnb. arXiv preprint. https://arxiv.org/abs/2112.09783 · arXiv
  16. Galton, F. (1907). Vox populi (the wisdom of crowds). Nature, 75(7), 450-451.
  17. Godel, W., Sanderson, Z., Aslett, K., Nagler, J., Bonneau, R., Persily, N., & Tucker, J. A. (2021). Moderating with the mob: Evaluating the efficacy of real-time crowdsourced fact-checking. Journal of Online Trust and Safety, 1(1). https://doi.org/10.54501/jots.v1i1.15 · DOI
  18. Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S. (2022, April). Jury learn- ing: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-19). Association for Computing Machinery. https://doi.org/10.1145/34911 02.3502004 · DOI
  19. Granovskiy, B., Gold, J. M., Sumpter, D. J., & Goldstone, R. L. (2015). Integration of social information by human groups. Topics in Cognitive Science, 7(3), 469-493.
  20. Graves, L. (2017). Anatomy of a fact check: Objective prac- tice and the contested epistemology of fact checking. Communication, Culture & Critique, 10(3), 518-537.
  21. Kao, A. B., Berdahl, A. M., Hartnett, A. T., Lutz, M. J., Bak- Coleman, J. B., Ioannou, C. C., Giam, X., & Couzin, I. D. (2018). Counteracting estimation bias and social influence to improve the wisdom of crowds. Journal of the Royal Society Interface, 15(141), Article 20180130.
  22. Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., Schudson, M., Sloman, S. A., Sunstein, C. R., Thorson, E. A., Watts, D. J., & Zittrain, J. L. (2018). The science of fake news. Science, 359(6380), 1094-1096.
  23. Le Mens, G., Kovács, B., Avrahami, J., & Kareev, Y. (2018). How endogenous crowd formation undermines the wis- dom of the crowd in online ratings. Psychological Science, 29(9), 1475-1490.
  24. Lim, C. (2018). Checking how fact-checkers check. Research & Politics, 5(3). https://doi.org/10.1177/2053168018786848 · DOI
  25. Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, USA, 108(22), 9020-9025.
  26. Massa, P., & Avesani, P. (2007). Trust metrics on controver- sial users: Balancing between tyranny of the majority. International Journal on Semantic Web and Information Systems (IJSWIS), 3(1), 39-64.
  27. Mena, P. (2020). Cleaning up social media: The effect of warning labels on likelihood of sharing false news on Facebook. Policy & Internet, 12(2), 165-183.
  28. Mitra, T., & Gilbert, E. (2015). Credbank: A large-scale social media corpus with associated credibility annotations. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 9, No. 1, pp. 258-267). AAAI Press. https://doi.org/10.1609/icwsm.v9i1.14625 · DOI
  29. Movshovitz-Attias, D., Movshovitz-Attias, Y., Steenkiste, P., & Faloutsos, C. (2013, August). Analysis of the reputation system and user contributions on a question answering website: Stackoverflow. In 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013) (pp. 886-893). IEEE. https://doi .org/10.1145/2492517.2500242 · DOI · Paper page
  30. Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., & Sigman, M. (2018). Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behaviour, 2(2), 126-132.
  31. Pan, C. A., Yakhmi, S., Iyer, T. P., Strasnick, E., Zhang, A. X., & Bernstein, M. S. (2022). Comparing the perceived legiti- macy of content moderation processes: Contractors, algo- rithms, expert panels, and digital juries. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW1), 1-31.
  32. Pennycook, G., Bear, A., Collins, E. T., & Rand, D. G. (2020). The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings. Management Science, 66(11), 4944-4957.
  33. Porter, E., & Wood, T. J. (2022). Political misinformation and fac- tual corrections on the Facebook news feed: Experimental evidence. The Journal of Politics, 84(3), 1812-1817.
  34. Pröllochs, N. (2022). Community-based fact-checking on Twitter's Birdwatch Platform. arXiv preprint. https://arxiv .org/abs/2104.07175
  35. Resnick, P., Alfayez, A., Im, J., & Gilbert, E. (2022). Informed crowds can effectively identify misinformation. arXiv pre- print. https://arxiv.org/abs/2108.07898 · arXiv
  36. Roitero, K., Soprano, M., Fan, S., Spina, D., Mizzaro, S., & Demartini, G. (2020, July). Can the crowd identify mis- information objectively? The effects of judgment scale and assessor's background. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 439-448). Association for Computing Machinery. https://doi.org/10 .1145/3397271.3401112 · DOI
  37. Saeed, M., Traub, N., Nicolas, M., Demartini, G., & Papotti, P. (2022, October). Crowdsourced fact-checking at Twitter: How does the crowd compare with experts? In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (pp. 1736-1746). Association for Computing Machinery. https://doi.org/ 10.1145/3511808.3557279 · DOI
  38. Satopää, V. A., Salikhov, M., Tetlock, P. E., & Mellers, B. (2023). Decomposing the effects of crowd-wisdom aggregators: The bias-information-noise (BIN) model. International Journal of Forecasting, 39(1), 470-485.
  39. Shi, F., Teplitskiy, M., Duede, E., & Evans, J. A. (2019). The wisdom of polarized crowds. Nature Human Behaviour, 3(4), 329-336.
  40. Silver, I., Mellers, B. A., & Tetlock, P. E. (2021). Wise team- work: Collective confidence calibration predicts the effec- tiveness of group discussion. Journal of Experimental Social Psychology, 96, Article 104157.
  41. Sorokin, A., & Forsyth, D. (2008, June). Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (pp. 1-8). IEEE.
  42. Toyokawa, W., Whalen, A., & Laland, K. N. (2019). Social learn- ing strategies regulate the wisdom and madness of inter- active crowds. Nature Human Behaviour, 3(2), 183-193.
  43. Wojcik, S., Hilgard, S., Judd, N., Mocanu, D., Ragain, S., Hunzaker, M. B., Coleman, K., & Baxter, J. (2022). Birdwatch: Crowd wisdom and bridging algorithms can inform understanding and reduce the spread of misinformation. arXiv preprint. https://arxiv.org/pdf/2210.15723.pdf · arXiv
  44. Yaqub, W., Kakhidze, O., Brockman, M. L., Memon, N., & Patil, S. (2020, April). Effects of credibility indicators on social media news sharing intent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14). Association for Comput- ing Machinery. https://doi.org/10.1145/3313831.337 6213 · DOI · Paper page
  45. Yasseri, T., & Menczer, F. (2021). Can crowdsourcing rescue the social marketplace of ideas? arXiv preprint. https:// arxiv.org/abs/2104.13754 · arXiv
  46. Zhang, Y. (2020). Identify experts through revealed confidence: Application to wisdom of crowds [Doctoral dissertation, Massachusetts Institute of Technology]. DSpace@MIT. https://dspace.mit.edu/bitstream/handle/1721.1/129085/ 1227097094-MIT.pdf?sequence=1&isAllowed=y
  47. Zhao, A., & Naaman, M. (2023). Variety, velocity, veracity, and viability: Evaluating the contributions of crowdsourced and professional fact-checking. SocArXiv preprint. https:// osf.io/preprints/socarxiv/yfxd3/

Cited by

Sorted by citations

Authors
Martel, C., Allen, J., Pennycook, G., & Rand, D. G.
Affiliations
  • Massachusetts Institute of Technology (US)
  • University of Regina (CA)
Corresponding author
Martel, Cameron
ORCID
Venue
Perspectives on Psychological Science
Publisher
SAGE Publications
Published
2023-08-18
Volume
19
Issue
2
Pages
477-488
ISSN
  • 1745-6916 (ISSN-L)
  • 1745-6924
DOI
10.1177/17456916231190388
OpenAlex
W4385969476
Cited by
84
References
12
Citations by year
2026: 24 · 2025: 25 · 2024: 25 · 2023: 8 · 2022: 2
FWCI
35.24
Type
journal article
Language
en
Open access
Licence
www.sagepub.com/licence-information-for-chorus (vor, 366 d)
Locations
  • Perspectives on Psychological Science (publishedVersion)
  • PubMed (publishedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Funders
  • William and Flora Hewlett Foundation
  • National Science Foundation Graduate Research Fellowship (174530)
  • National Science Foundation (FAIN 2047152)
  • TDF Foundation
  • Office of Naval Research (N00014-21-2573)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Arts and Humanities › Philosophy › Epistemology, Ethics, and Metaphysics
Keywords
misinformation, crowdsourcing, wisdom of crowds, crowdsourced fact-checking, lay judgment, fact-checking, news quality, misinformation detection, social media
MeSH terms
Communication, Humans, Judgment, Social Media, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review