Scaling up fact-checking using the wisdom of crowds
Open accessAbstract
When rating articles’ accuracy, a small politically balanced crowd of laypeople yields high agreement with fact-checkers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Community Notes achieves high identification accuracy enabled by the wisdom of crowds.
“and (ii) it achieves high identification accuracy enabled by the wisdom of crowds (Drolsbach and Pr¨ ollochs 2023a; Allen et al. 2021; Martel et al. 2024)”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
The Discussion (page unavailable) reports that a small, politically balanced crowd identified misinformation with performance matching professional fact-checkers. But the study tested people assigned specific articles to rate, not Community Notes; the Discussion explicitly says it is unclear whether the findings generalize to Twitter’s Birdwatch, where participants choose what to rate. It therefore supports the wisdom-of-crowds mechanism, not the claim about Community Notes’ identification accuracy. (LLM, conf=0.90)
Experimental — accuracy not yet measuredCrowd-based approaches can achieve accuracy levels comparable to experts and overcome scalability issues.
“Yet, evidence from studies on traditional misinformation shows that crowd-based approaches, where multiple judgments are aggregated, can achieve accuracy levels comparable to those of experts and overcome scalability issues (Allen et al. 2021; Martel et al. 2024; Drolsbach and Pr¨ ollochs 2023a; He et al. 2025; Pilarski and Pr¨ ollochs 2026).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
Averaged responses of ordinary people are on par with third-party fact-checkers for both factual claims in articles and overall accuracy of content from URL domain names.
“Recent empirical research finds that averaged responses of ordinary people are on par with third-party fact-checkers for both factual claims in articles (47) and overall accuracy of content from URL domain names (48, 49).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256 · see citing works
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160 · see citing works
Self-citationFake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304 · see citing works
When Corrections Fail: The Persistence of Political Misperceptions
Nyhan, B., & Reifler, J. (2010) · Political Behavior · doi:10.1007/s11109-010-9112-2 · Cited by 3,173 · see citing works
Misinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920 · see citing works
The Polarizing Impact of Science Literacy and Numeracy on Perceived Climate Change Risks
Braman, D., Kahan, D. M., Peters, E. M., Wittlin, M., Slovic, P., Ouellette, L. L., & Mandel, G. N. (2012) · Scholarly Commons - George Washington University Law School (George Washington University) · Cited by 2,299 · see citing works
The Persuasiveness of Source Credibility: A Critical Review of Five Decades' Evidence
Pornpitakpan, C. (2004) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2004.tb02547.x · Cited by 2,205 · cited 1× · see citing works
“While a large literature has shown that people preferentially believe information from trusted sources [for a review, see ( 52 )], this work has usually focused on information shared by people who are more or less trusted.”
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148 · see citing works
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022 · see citing works
Self-citationPennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715 · see citing works
Self-citationPennycook, G., & Rand, D. G. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.02.007 · Cited by 1,360 · see citing works
Self-citationIdeology, motivated reasoning, and cognitive reflection
Kahan, D. M. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500005271 · Cited by 1,302 · see citing works
Prior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268 · see citing works
Self-citationShifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155 · see citing works
Self-citationGALTON, F. (1907) · Nature · doi:10.1038/075450a0 · Cited by 1,080 · see citing works
CSI: A Hybrid Deep Model for Fake News Detection
Ruchansky, N., Seo, S., & Liu, Y. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132877 · Cited by 1,039 · see citing works
Automatic deception detection: Methods for finding fake news
Conroy, N. K., Rubin, V. L., & Chen, Y. (2015) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2015.145052010082 · Cited by 990 · see citing works
Fighting misinformation on social media using crowdsourced judgments of news source quality
Pennycook, G., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806781116 · Cited by 947 · cited 5× · see citing works
“Second, publisher trust ratings are largely driven by familiarity: People overwhelmingly distrust news publishers that they are unfamiliar with ( 20 , 21 ).”
InfluentialSelf-citationThe Elusive Backfire Effect: Mass Attitudes’ Steadfast Factual Adherence
Wood, T., & Porter, E. (2019) · Political Behavior · doi:10.1007/s11109-018-9443-y · Cited by 904 · see citing works
Pennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879 · see citing works
Self-citationPennycook, G., Bear, A., Collins, E., & Rand, D. G. (2020) · Management Science · doi:10.1287/mnsc.2019.3478 · Cited by 718 · see citing works
Self-citationAutomatic Detection of Fake NewsPreprint
Pérez‐Rosas, V., Kleinberg, B., Lefevre, A., & Mihalcea, R. F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07104 · Cited by 709 · see citing works
Knowledge does not protect against illusory truth.
Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000098 · Cited by 598 · see citing works
Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines.
Bago, B., Rand, D. G., & Pennycook, G. (2020) · Journal of Experimental Psychology: General · doi:10.1037/xge0000729 · Cited by 593 · see citing works
Self-citationDeception detection for news: Three types of fakes
Rubin, V. L., Chen, Y., & Conroy, N. K. (2015) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2015.145052010083 · Cited by 558 · see citing works
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · see citing works
Assessing the Capacity of Mass Electorates
Converse, P. E. (2000) · Annual Review of Political Science · doi:10.1146/annurev.polisci.3.1.331 · Cited by 381 · see citing works
Searching for the backfire effect: Measurement and design considerations.
Swire-Thompson, B., DeGutis, J., & Lazer, D. (2020) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2020.06.006 · Cited by 360 · see citing works
Ideology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study
Kahan, D. M. (2012) · doi:10.2139/ssrn.2182588 · Cited by 331 · see citing works
Bronstein, M. V., Pennycook, G., Bear, A., Rand, D. G., & Cannon, T. D. (2019) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2018.09.005 · Cited by 324 · see citing works
Self-citationA note on aggregating opinions
Hogarth, R. M. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90037-5 · Cited by 322 · see citing works
The Epistemology of Fact Checking
Uscinski, J. E., & Butler, R. W. (2013) · Critical Review · doi:10.1080/08913811.2013.843872 · Cited by 308 · see citing works
Mannes, A. E., Soll, J. B., & Larrick, R. P. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036677 · Cited by 265 · see citing works
Anatomy of a Fact Check: Objective Practice and the Contested Epistemology of Fact Checking
Graves, L. (2016) · Communication Culture and Critique · doi:10.1111/cccr.12163 · Cited by 238 · see citing works
Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation
Kim, J., Tabibian, B., Oh, A., Schölkopf, B., & Gomez-Rodriguez, M. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159734 · Cited by 230 · see citing works
Gabielkov, M., Ramachandran, A., Chaintreau, A., & Legout, A. (2016) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/2896377.2901462 · Cited by 219 · see citing works
Social learning and partisan bias in the interpretation of climate trends
Guilbeault, D., Becker, J., & Centola, D. (2018) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1722664115 · Cited by 188 · see citing works
Harnessing the Wisdom of Crowds
Da, Z., & Huang, X. (2019) · Management Science · doi:10.1287/mnsc.2019.3294 · Cited by 183 · see citing works
Effects of Credibility Indicators on Social Media News Sharing Intent
Yaqub, W., Kakhidze, O., Brockman, M. L., Memon, N., & Patil, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376213 · Cited by 171 · see citing works
Checking how fact-checkers check
Lim, C. (2018) · Research & Politics · doi:10.1177/2053168018786848 · Cited by 166 · see citing works
The wisdom of polarized crowds
Shi, F., Teplitskiy, M., Duede, E., & Evans, J. A. (2019) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · Cited by 162 · see citing works
Becker, J. A., Porter, E. V., & Centola, D. M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1817195116 · Cited by 157 · see citing works
Confirmation bias with motivated beliefs
Charness, G., & Dave, C. (2017) · Games and Economic Behavior · doi:10.1016/j.geb.2017.02.015 · Cited by 154 · see citing works
Fighting misperceptions and doubting journalists’ objectivity: A review of fact-checking literature
Nieminen, S., & Rapeli, L. (2019) · Political Studies Review · doi:10.1177/1478929918786852 · Cited by 150 · see citing works
Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media
Dias, N., Pennycook, G., & Rand, D. G. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-001 · Cited by 145 · see citing works
Self-citationThe wisdom of polarized crowds
Shi, F., Teplitskiy, M., Duede, E., & Evans, J. A. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0541-6 · Cited by 129 · see citing works
Ecker, U. K. H., Lewandowsky, S., & Chadwick, M. (2020) · Cognitive Research Principles and Implications · doi:10.1186/s41235-020-00241-6 · Cited by 117 · see citing works
Checking the Fact-Checkers in 2008: Predicting Political Ad Scrutiny and Assessing Consistency
Amazeen, M. A. (2014) · Journal of Political Marketing · doi:10.1080/15377857.2014.959691 · Cited by 91 · see citing works
The wisdom of smaller, smarter crowds
Goldstein, D. G., McAfee, R. P., & Suri, S. (2014) · doi:10.1145/2600057.2602886 · Cited by 77 · see citing works
Investigating Differences in Crowdsourced News Credibility Assessment
Bhuiyan, M. M., Zhang, A. X., Sehat, C. M., & Mitra, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415164 · Cited by 75 · see citing works
Epstein, Z., Pennycook, G., & Rand, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376232 · Cited by 67 · see citing works
Self-citationEmphasizing publishers does not effectively reduce susceptibility to misinformation on social mediaPreprint
Dias, N., Pennycook, G., & Rand, D. G. (2020) · doi:10.31234/osf.io/zv4dq · Cited by 60 · see citing works
Self-citationThe Wisdom of Crowds of Doctors
Kattan, M. W., O’Rourke, C., Yu, C., & Chagin, K. M. (2015) · Medical Decision Making · doi:10.1177/0272989x15581615 · Cited by 44 · see citing works
The Role of Source, Headline and Expressive Responding in Political News Evaluation
Jakesch, M., Koren, M., Evtushenko, A., & Naaman, M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3306403 · Cited by 30 · see citing works
Keck, S., & Tang, W. (2020) · Psychological Science · doi:10.1177/0956797620941840 · Cited by 25 · see citing works
Thought as a determinant of political opinion
Sloman, S. A., & Rabb, N. (2019) · Cognition · doi:10.1016/j.cognition.2019.02.014 · Cited by 19 · see citing works
Informed Crowds Can Effectively Identify Misinformation.Preprint
Resnick, P., Alfayez, A., Im, J., & Gilbert, É. (2021) · arXiv (Cornell University) · Cited by 8 · see citing works
Principled Evaluation with Human Labels: One Rater at a Time and Rater EquivalencePreprint
Resnick, P., Kong, Y., Schoenebeck, G. R., & Weninger, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01254 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Pennycook, G., & Rand, D. G. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.02.007 · Cited by 1,360 · see citing works
Self-citationShifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155 · see citing works
Self-citationDeepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248 · see citing works
Spread of misinformation on social media: What contributes to it and how to combat it
Chen, S., Xiao, L., & Kumar, A. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107643 · Cited by 211 · see citing works
Understanding and combatting misinformation across 16 countries on six continents
Arechar, A. A., Allen, J., Berinsky, A. J., Cole, R., Epstein, Z., Garimella, K., Gully, A., Lu, J. G., Ross, R. M., Stagnaro, M. N., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01641-6 · Cited by 188 · see citing works
Self-citationQuantifying the impact of misinformation and vaccine-skeptical content on Facebook
Allen, J. N. L., Watts, D. J., & Rand, D. G. (2024) · Science · doi:10.1126/science.adk3451 · Cited by 154 · see citing works
Self-citationJury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M., Lam, M. S., Park, J.-S., Patel, K., Hancock, J. T., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 143 · see citing works
Dual emotion based fake news detection: A deep attention-weight update approach
Luvembe, A. M., Li, W., Li, S., Liu, F., & Xu, G. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103354 · Cited by 126 · see citing works
Online searches to evaluate misinformation can increase its perceived veracity
Aslett, K., Sanderson, Z., Godel, W., Persily, N., Nagler, J., & Tucker, J. A. (2023) · Nature · doi:10.1038/s41586-023-06883-y · Cited by 118 · see citing works
Jahanbakhsh, F., Zhang, A. X., Berinsky, A. J., Pennycook, G., Rand, D. G., & Karger, D. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449092 · Cited by 112 · see citing works
Self-citationDo social media undermine social cohesion? A critical review
González-Bailón, S., & Lelkes, Y. (2022) · Social Issues and Policy Review · doi:10.1111/sipr.12091 · Cited by 110 · see citing works
Altay, S., de Araujo, E., & Mercier, H. (2021) · Digital Journalism · doi:10.1080/21670811.2021.1941163 · Cited by 97 · see citing works
Human Cooperation and the Crises of Climate Change, COVID-19, and Misinformation
Lange, P. A. M. V., & Rand, D. G. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-110044 · Cited by 98 · see citing works
Self-citationRuffo, G., Semeraro, A., Giachanou, A., & Rosso, P. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100531 · Cited by 110 · see citing works
Liu, X., Li, Q. L., Wang, L., & Metzger, M. J. (2023) · Communication Research · doi:10.1177/00936502231206419 · Cited by 92 · see citing works
Tackling misinformation: What researchers could do with social media data
Pasquetto, I. V., Swire‐Thompson, B., & Amazeen, M. A. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-49 · Cited by 86 · see citing works
Crowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84 · see citing works
Self-citationNudging Social Media toward Accuracy
Pennycook, G., & Rand, D. G. (2022) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/00027162221092342 · Cited by 106 · see citing works
Self-citationMisinformation exploits outrage to spread online
McLoughlin, K. L., Brady, W. J., Goolsbee, A., Kaiser, B., Klonick, K., & Crockett, M. J. (2024) · Science · doi:10.1126/science.adl2829 · Cited by 81 · see citing works
The state of human-centered NLP technology for fact-checking
Das, A., Liu, H., Kovatchev, V., & Lease, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103219 · Cited by 74 · see citing works
True or False: Studying the Work Practices of Professional Fact-Checkers
Micallef, N., Armacost, V., Memon, N., & Patil, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512974 · Cited by 70 · see citing works
Community-Based Fact-Checking on Twitter’s Birdwatch Platform
Pröllochs, N. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19335 · Cited by 68 · see citing works
High level of correspondence across different news domain quality rating sets
Lin, H., Lasser, J., Lewandowsky, S., Cole, R., Gully, A., Rand, D. G., & Pennycook, G. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad286 · Cited by 75 · see citing works
Self-citationModerating with the Mob: Evaluating the Efficacy of Real-Time Crowdsourced Fact-Checking
Godel, W., Sanderson, Z., Aslett, K., Nagler, J., Bonneau, R., Persily, N., & Tucker, J. A. (2021) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i1.15 · Cited by 61 · see citing works
Can Online Juries Make Consistent, Repeatable Decisions?
Hu, X. E., Whiting, M. E., & Bernstein, M. S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445433 · Cited by 60 · see citing works
Do Explanations Increase the Effectiveness of AI-Crowd Generated Fake News Warnings?
Epstein, Z., Foppiani, N., Hilgard, S., Sharma, S., Glassman, E. L., & Rand, D. G. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19283 · Cited by 61 · see citing works
Self-citationCommunity notes increase trust in fact-checking on social media
Drolsbach, C., Solovev, K., & Pröllochs, N. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae217 · Cited by 59 · see citing works
Leveraging ChatGPT for Efficient Fact-CheckingPreprint
Hoes, E., Altay, S., & Bermeo, J. (2023) · doi:10.31234/osf.io/qnjkf · Cited by 61 · see citing works
Displaying News Source Trustworthiness Ratings Reduces Sharing Intentions for False News Posts
Celadin, T., Capraro, V., Pennycook, G., & Rand, D. G. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i5.100 · Cited by 50 · see citing works
Self-citationSpotting false news and doubting true news: a systematic review and meta-analysis of news judgements
Pfänder, J., & Altay, S. (2025) · Nature Human Behaviour · doi:10.1038/s41562-024-02086-1 · Cited by 71 · see citing works
Austin, E. W., Borah, P., & Domgaard, S. (2021) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-58 · Cited by 50 · see citing works
He, B., Ahamad, M., & Kumar, S. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583388 · Cited by 49 · see citing works
Lateral reading and monetary incentives to spot disinformation about science
Panizza, F., Ronzani, P., Martini, C. M. C., Mattavelli, S., Morisseau, T., & Motterlini, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-09168-y · Cited by 47 · see citing works
Changing the incentive structure of social media platforms to halt the spread of misinformation
Globig, L. K., Holtz, N., & Sharot, T. (2023) · eLife · doi:10.7554/elife.85767 · Cited by 50 · see citing works
Chen, Y.-P., Chen, Y., Yang, K.-C., Lai, F., Huang, C., Chen, Y.-N., & Tu, Y.-C. (2022) · Journal of Medical Internet Research · doi:10.2196/36830 · Cited by 40 · see citing works
Differences in misinformation sharing can lead to politically asymmetric sanctions
Mosleh, M., Yang, Q., Zaman, T. R., Pennycook, G., & Rand, D. G. (2024) · Nature · doi:10.1038/s41586-024-07942-8 · Cited by 35 · see citing works
Self-citationHow Effective Are Interventions Against Misinformation?
Altay, S. (2022) · doi:10.31234/osf.io/sm3vk · Cited by 32 · see citing works
Lesher, M., Pawelec, H., & Desai, A. (2022) · OECD going digital toolkit notes · doi:10.1787/84b62df1-en · Cited by 31 · see citing works
In generative AI we trust: can chatbots effectively verify political information?
Kuznetsova, E., Makhortykh, M., Vziatysheva, V., Stolze, M., Baghumyan, A., & Urman, A. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00338-8 · Cited by 32 · see citing works
Fact-checking in the age of AI: Reducing biases with non-human information sources
Moon, W.-K., & Kahlor, L. A. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102760 · Cited by 28 · see citing works
van der Linden, S., Albarracín, D., Fazio, L. K., Freelon, D., Roozenbeek, J., Swire‐Thompson, B., & Bavel, J. J. V. (2023) · PsycEXTRA Dataset · doi:10.1037/e506432023-001 · Cited by 28 · see citing works
Understanding and Combating Misinformation Across 16 Countries on Six Continents
Arechar, A. A., Allen, J. N. L., Berinsky, A. J., Cole, R., Epstein, Z., Garimella, K., Gully, A., Lu, J. G., Ross, R. M., Stagnaro, M., et al. (2022) · doi:10.31234/osf.io/a9frz · Cited by 25 · see citing works
Self-citationCommunity Notes vs. Snoping: How the Crowd Selects Fact-Checking Targets on Social Media
Pilarski, M., Solovev, K., & Pröllochs, N. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31387 · Cited by 26 · see citing works
Ma, H. (., Huang, W. W., & Dennis, A. R. (2024) · Journal of Management Information Systems · doi:10.1080/07421222.2024.2376381 · Cited by 27 · see citing works
Crowdsourced Fact-checking: Does It Actually Work?
Barbera, D. L., Maddalena, E., Soprano, M., Roitero, K., Demartini, G., Ceolin, D., Spina, D., & Mizzaro, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103792 · Cited by 25 · see citing works
On the Efficacy of Accuracy Prompts Across Partisan Lines: An Adversarial Collaboration
Martel, C., Rathje, S., Clark, C. J., Pennycook, G., Bavel, J. J. V., Rand, D. G., & van der Linden, S. (2024) · Psychological Science · doi:10.1177/09567976241232905 · Cited by 25 · see citing works
Self-citationMahl, D., Zeng, J., Schäfer, M. S., Egert, F. A., & de Oliveira, T. M. (2024) · The International Journal of Press/Politics · doi:10.1177/19401612241270004 · Cited by 24 · see citing works
Community notes reduce engagement with and diffusion of false information online
Slaughter, I., Peytavin, A., Ugander, J. H., & Saveski, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2503413122 · Cited by 23 · see citing works
Can Crowdsourcing Rescue the Social Marketplace of Ideas?
Yasseri, T., & Menczer, F. (2023) · Communications of the ACM · doi:10.1145/3578645 · Cited by 22 · see citing works
Neumann, T., De‐Arteaga, M., & Fazelpour, S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533205 · Cited by 23 · see citing works
DelphiAgent: A trustworthy multi-agent verification framework for automated fact verification
Xiong, C. C., Zheng, G., Ma, X., Li, C., & Zeng, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104241 · Cited by 22 · see citing works
Untangling the web of misinformation and false beliefs
Johar, G. V. (2022) · Journal of Consumer Psychology · doi:10.1002/jcpy.1288 · Cited by 21 · see citing works
High level of correspondence across different news domain quality rating setsPreprint
Lin, H., Lasser, J., Lewandowsky, S., Cole, R., Gully, A., Rand, D. G., & Pennycook, G. (2022) · doi:10.31234/osf.io/qy94s · Cited by 20 · see citing works
Self-citationBerger, L., Kerkhof, A., Mindl, F., & Münster, J. (2025) · Journal of Public Economics · doi:10.1016/j.jpubeco.2025.105345 · Cited by 32 · see citing works
Markowitz, D. M., Levine, T. R., Serota, K. B., & Moore, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289004 · Cited by 19 · see citing works
Lisnic, M., Lex, A., & Kogan, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642448 · Cited by 19 · see citing works
Johansson, P., Enoch, F., Hale, S. A., Vidgen, B., Bereskin, C., Margetts, H. Z., & Bright, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4648332 · Cited by 24 · see citing works
Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting
Blanchard, M. D., Herzog, S. M., Kämmer, J. E., Zöller, N., Kostopoulou, O., & Kurvers, R. H. J. M. (2024) · Medical Decision Making · doi:10.1177/0272989x241241001 · Cited by 15 · see citing works
Diverse misinformation: impacts of human biases on detection of deepfakes on networks
Lovato, J., St-Onge, J., Harp, R., Salazar Lopez, G., Rogers, S. P., Haq, I. U., Hébert-Dufresne, L., & Onaolapo, J. (2024) · npj Complexity · doi:10.1038/s44260-024-00006-y · Cited by 16 · see citing works
Akhtar, M. M., Masood, R., Ikram, M., & Kanhere, S. S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3644998 · Cited by 16 · see citing works
Sincere or Motivated? Partisan Bias in Advice-taking
Zhang, Y., & Rand, D. G. (2021) · Judgment and Decision Making · doi:10.1017/jdm.2023.28 · Cited by 15 · see citing works
Self-citation'Debunk-It-Yourself': Health Professionals Strategies for Responding to Misinformation on TikTok
Sharevski, F., Loop, J. V., Jachim, P., Devine, A., & Das, S. (2024) · New Security Paradigms Workshop · doi:10.1145/3703465.3703469 · Cited by 15 · see citing works
Believability and Harmfulness Shape the Virality of Misleading Social Media Posts
Drolsbach, C., & Pröllochs, N. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583857 · Cited by 15 · see citing works
Resnick, P., Alfayez, A., Im, J., & Gilbert, É. (2023) · Collective Intelligence · doi:10.1177/26339137231173407 · Cited by 14 · see citing works
Shin, J., Lewis, S. C., Kim, S., & Thorson, K. (2024) · New Media & Society · doi:10.1177/14614448241228851 · Cited by 14 · see citing works
He, B., Hu, Y., Lee, Y.-C., Oh, S., Verma, G., & Kumar, S. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3694980 · Cited by 14 · see citing works
Corsi, G., Seger, E., & hÉigeartaigh, S. Ó. (2024) · Online Social Networks and Media · doi:10.1016/j.osnem.2024.100291 · Cited by 14 · see citing works
Wirtschafter, V., & Majumder, S. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v2i1.139 · Cited by 13 · see citing works
References to unbiased sources increase the helpfulness of community fact-checks
Solovev, K., & Pröllochs, N. (2025) · Scientific Reports · doi:10.1038/s41598-025-09372-6 · Cited by 12 · see citing works
CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation
Venkatagiri, S., Mukhopadhyay, A., Hicks, D. L., Brantly, A. F., & Luther, K. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3595997 · Cited by 12 · see citing works
How do online users respond to crowdsourced fact-checking?
Panizza, F., Ronzani, P., Morisseau, T., Mattavelli, S., & Martini, C. M. C. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02329-y · Cited by 13 · see citing works
Conversational Agents to Facilitate Deliberation on Harmful Content in WhatsApp Groups
Agarwal, D., Shahid, F., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687030 · Cited by 15 · see citing works
Combining Human and Machine Confidence in Truthfulness Assessment
Qu, Y., Roitero, K., Barbera, D. L., Spina, D., Mizzaro, S., & Demartini, G. (2022) · Journal of Data and Information Quality · doi:10.1145/3546916 · Cited by 12 · see citing works
The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk Evaluation
Duan, X., Ho, C.-J., & Yin, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512239 · Cited by 12 · see citing works
Collaboratively adding context to social media posts reduces the sharing of false news.Preprint
Renault, T., Amariles, D. R., & Troussel, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4800565 · Cited by 15 · see citing works
Combining Large Language Models and Crowdsourcing for Hybrid Human-AI Misinformation Detection
Zeng, X., Barbera, D. L., Roitero, K., Zubiaga, A., & Mizzaro, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657965 · Cited by 12 · see citing works
Nudging social media sharing towards accuracyPreprint
Pennycook, G., & Rand, D. G. (2021) · doi:10.31234/osf.io/tp6vy · Cited by 10 · see citing works
Self-citationConsidering information-sharing motives to reduce misinformation
Globig, L. K., & Sharot, T. (2024) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2024.101852 · Cited by 10 · see citing works
Measuring receptivity to misinformation at scale on a social media platform
Tokita, C. K., Aslett, K., Godel, W., Sanderson, Z., Tucker, J. A., Nagler, J., Persily, N., & Bonneau, R. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae396 · Cited by 10 · see citing works
Matamoros-Fernández, A., & Jude, N. (2025) · New Media & Society · doi:10.1177/14614448251314399 · Cited by 11 · see citing works
When More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21994 · Cited by 9 · see citing works
Miyazaki, K., Uchiba, T., Tanaka, K., An, J., Kwak, H., & Sasahara, K. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22176 · Cited by 10 · see citing works
Interactive crowdsourcing to fact-check politicians.
Espina-Mairal, S., Bustos, F., Solovey, G., & Navajas, J. (2023) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000492 · Cited by 10 · see citing works
Community notes reduce the spread of misleading posts on XPreprint
Chuai, Y., Pilarski, M., Lenzini, G., & Pröllochs, N. (2024) · doi:10.31219/osf.io/3a4fe · Cited by 10 · see citing works
(2021) · doi:10.31234/osf.io/ar96c · Cited by 12 · see citing works
Self-citationHasan, E., Duhaime, E., & Trueblood, J. S. (2024) · Cognitive Research Principles and Implications · doi:10.1186/s41235-024-00558-6 · Cited by 9 · see citing works
Spotting False News and Doubting True News: A Meta-Analysis of News JudgmentsPreprint
Pfänder, J., & Altay, S. (2023) · doi:10.31219/osf.io/n9h4y · Cited by 9 · see citing works
Stewart, A. J., Arechar, A. A., Rand, D. G., & Plotkin, J. B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2315195121 · Cited by 8 · see citing works
Self-citationKim, J., Wang, Z., Shi, H., Ling, H.-K., & Evans, J. A. (2025) · Nature Communications · doi:10.1038/s41467-025-55868-0 · Cited by 8 · see citing works
Abels, C. M., Lopez-Lopez, E., Burton, J. W., Holford, D. L., Brinkmann, L. U. L., Herzog, S. M., & Lewandowsky, S. (2025) · Behavioral Science & Policy · doi:10.1177/23794607251347020 · Cited by 8 · see citing works
Jornalistas Locais e Fact-Checking: Um Estudo Exploratório em Portugal e Espanha
Jerónimo, P., & Esparza, M. S. (2023) · Comunicação e Sociedade · doi:10.17231/comsoc.44(2023).4553 · Cited by 8 · see citing works
The unappreciated role of intent in algorithmic moderation of abusive content on social media
Wang, X., Koneru, S., Venkit, P. N., Frischmann, B. M., & Rajtmajer, S. M. (2025) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-180 · Cited by 14 · see citing works
Emotional language reduces belief in false claims
Phillips, S. C., Wang, S. Y. N., Carley, K. M., Rand, D. G., & Pennycook, G. (2025) · Judgment and Decision Making · doi:10.1017/jdm.2025.10019 · Cited by 7 · see citing works
Self-citationCommunity-based fact-checking reduces the spread of misleading posts on X (formerly Twitter)
Chuai, Y., Pilarski, M., Renault, T., Amariles, D. R., Troussel, A., Lenzini, G., & Pröllochs, N. (2026) · Nature Communications · doi:10.1038/s41467-026-72597-0 · Cited by 7 · see citing works
Realtime user ratings as a strategy for combatting misinformation: an experimental study
Stein, J., Frey, V., & van de Rijt, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-28597-x · Cited by 7 · see citing works
Yaqub, W., Kay, J., & Goldwater, M. B. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637335 · Cited by 7 · see citing works
Unbiased misinformation policies sanction conservatives more than liberalsPreprint
Mosleh, M., Yang, Q., Zaman, T. R., Pennycook, G., & Rand, D. G. (2022) · doi:10.31234/osf.io/ay9q5 · Cited by 7 · see citing works
Self-citationNewsComp: Facilitating Diverse News Reading through Comparative Annotation
Bhuiyan, M. M., Lee, S. W., Goyal, N., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581244 · Cited by 6 · see citing works
Yours Truly: A Credibility Framework for Effortless LLM-Powered Fact Checking
Krishnamurthy, V., & Balaji, V. (2024) · IEEE Access · doi:10.1109/access.2024.3520187 · Cited by 6 · see citing works
Limited effectiveness of psychological inoculation against misinformation in a social media feed
Wang, S.-Y. N., Phillips, S. C., Carley, K. M., Lin, H., & Pennycook, G. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf172 · Cited by 7 · see citing works
Self-citation
- Authors
- Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Nottingham (GB)
- University of Regina (CA)
- Corresponding author
- Allen, Jennifer
- Arechar, Antonio A.
- Rand, David G.
- Venue
- Science Advances
- Published
- 2021-09-03
- Volume
- 7
- Issue
- 36
- Pages
- eabf4393-eabf4393
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC8442902
- OpenAlex
- W3172233798
- Cited by
- 234
- References
- 56
- Citations by year
- 2026: 44 · 2025: 47 · 2024: 48 · 2023: 45 · 2022: 33 · 2021: 15 · 2020: 2
- FWCI
- 49.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- PsyArXiv (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation (61061)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- fact-checking, misinformation, crowdsourcing, wisdom of crowds, laypeople, political balance, Facebook, news headlines, cognitive reflection, political knowledge, partisanship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
