The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Scaling up fact-checking using the wisdom of crowds
Open accessAbstract
When rating articles’ accuracy, a small politically balanced crowd of laypeople yields high agreement with fact-checkers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and (ii) it achieves high identification accuracy enabled by the wisdom of crowds (Drolsbach and Pr¨ ollochs 2023a; Allen et al. 2021; Martel et al. 2024)”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Yet, evidence from studies on traditional misinformation shows that crowd-based approaches, where multiple judgments are aggregated, can achieve accuracy levels comparable to those of experts and overcome scalability issues (Allen et al. 2021; Martel et al. 2024; Drolsbach and Pr¨ ollochs 2023a; He et al. 2025; Pilarski and Pr¨ ollochs 2026).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Allen et al. 2021
In the citing paper
“Recent empirical research finds that averaged responses of ordinary people are on par with third-party fact-checkers for both factual claims in articles (47) and overall accuracy of content from URL domain names (48, 49).”
From Deepfake detection by human crowds, machines, and machine-informed crowds (2022) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Self-citationFake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304
When Corrections Fail: The Persistence of Political Misperceptions
Nyhan, B., & Reifler, J. (2010) · Political Behavior · doi:10.1007/s11109-010-9112-2 · Cited by 3,173 · cited 1×
“Debunking false claims typically reduces incorrect belief ( 5 ) [early concerns about potential “backfire effects” ( 6 , 7 ) have not been supported by subsequent work; for a review, see ( 8 )].”
Misinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920
The Polarizing Impact of Science Literacy and Numeracy on Perceived Climate Change Risks
Braman, D., Kahan, D. M., Peters, E. M., Wittlin, M., Slovic, P., Ouellette, L. L., & Mandel, G. N. (2012) · Scholarly Commons - George Washington University Law School (George Washington University) · Cited by 2,421
The Persuasiveness of Source Credibility: A Critical Review of Five Decades' Evidence
Pornpitakpan, C. (2004) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2004.tb02547.x · Cited by 2,205 · cited 1×
“While a large literature has shown that people preferentially believe information from trusted sources [for a review, see ( 52 )], this work has usually focused on information shared by people who are more or less trusted.”
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022 · cited 4×
“…we cannot rule out this possibility, we did replicate our results by analyzing a previously published dataset of researcher-selected headlines ( 25 ) and found that the crowd performed substantially better than for the articles provided by Facebook (see section S16), suggesting that the…”
InfluentialSelf-citationPennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715 · cited 2×
“…with rapidly evolving facts, such as in the case of the coronavirus disease 2019 (COVID-19) news environment, results for both the crowd and fact-checkers would differ [although studies with researcher-selected true and false COVID-19 headlines find high crowd performance, e.g., ( 39 )].”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- 10.1038/nclimate1547 · DOI · Paper page
Cited by
Sorted by citations
Pennycook, G., & Rand, D. G. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.02.007 · Cited by 1,360
“First, truth is not a black-and-white, clearly defined property: even professional fact-checkers often disagree on how exactly to classify content [77,78].”
BackgroundResultInfluentialSelf-citationShifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155
Self-citationDeepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248
Spread of misinformation on social media: What contributes to it and how to combat it
Chen, S., Xiao, L., & Kumar, A. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107643 · Cited by 239
Understanding and combatting misinformation across 16 countries on six continents
Arechar, A. A., Allen, J., Berinsky, A. J., Cole, R., Epstein, Z., Garimella, K., Gully, A., Lu, J. G., Ross, R. M., Stagnaro, M. N., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01641-6 · Cited by 188
Self-citationQuantifying the impact of misinformation and vaccine-skeptical content on Facebook
Allen, J. N. L., Watts, D. J., & Rand, D. G. (2024) · Science · doi:10.1126/science.adk3451 · Cited by 173
“…with the previously discussed crowdsourced ratings of (i) whether the headline suggests that vaccines are harmful or helpful to health and (ii) headline accuracy [which, consistent with prior work ( 52 – 54 ), agreed highly with expert judgments; correlation coefficient ( r ) = 0.72, P < 0.000001].”
BackgroundInfluentialSelf-citationJury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M., Lam, M. S., Park, J.-S., Patel, K., Hancock, J. T., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 179
“To see how this might be true, we can examine annotator disagreement rates in today’s datasets: for instance, in a toxicity task, over one third of annotators on average disagree with any toxic classification, even after accounting for label noise [37]; in a misinformation classification task, three professional fact…”
BackgroundDual emotion based fake news detection: A deep attention-weight update approach
Luvembe, A. M., Li, W., Li, S., Liu, F., & Xu, G. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103354 · Cited by 126
Online searches to evaluate misinformation can increase its perceived veracity
Aslett, K., Sanderson, Z., Godel, W., Persily, N., Nagler, J., & Tucker, J. A. (2023) · Nature · doi:10.1038/s41586-023-06883-y · Cited by 151
Jahanbakhsh, F., Zhang, A. X., Berinsky, A. J., Pennycook, G., Rand, D. G., & Karger, D. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449092 · Cited by 112
“Some of these initiatives include detecting false or misleading information using machine learning algorithms [12, 57, 67] and crowdsourcing [5, 9, 19, 33, 54], identifying bad actors and helping good actors differentiate themselves 1 [82], and providing fact-checked information related to…”
BackgroundSelf-citationDo social media undermine social cohesion? A critical review
González‐Bailón, S., & Yphtach Lelkes, B. (2022) · Social Issues and Policy Review · doi:10.1111/sipr.12091 · Cited by 140
Altay, S., de Araujo, E., & Mercier, H. (2021) · Digital Journalism · doi:10.1080/21670811.2021.1941163 · Cited by 120
Human Cooperation and the Crises of Climate Change, COVID-19, and Misinformation
Lange, P. A. M. V., & Rand, D. G. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-110044 · Cited by 103
Self-citationRuffo, G., Semeraro, A., Giachanou, A., & Rosso, P. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100531 · Cited by 129
“[10], which noted that a small but controlled crowd of non-experts rates news similarly to fact-checkers, but it is a disputed conclusion [115].”
BackgroundLiu, X., Qi, L., Wang, L., & Metzger, M. J. (2023) · Communication Research · doi:10.1177/00936502231206419 · Cited by 94
“Somewhat related, two recent studies found that people with a higher tendency toward analytic thinking were more likely to agree with corrective messages than those lower in analytic thinking (Allen et al., 2021; Martel et al., 2021).”
Tackling misinformation: What researchers could do with social media data
Pasquetto, I. V., Swire‐Thompson, B., & Amazeen, M. A. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-49 · Cited by 99
“Finally, we would also be interested in assessing the impact of incorporating layperson (user) accuracy ratings, either of news sources (Epstein et al, 2020; Pennycook & Rand, 2019) or individual pieces of content (Allen, Arechar, et al. 2020), into social media ranking algorithms.”
Crowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
“Our reanalysis and Allen et al. (2021) also focused on the relative agreement between crowds and fact-checkers, whereas Godel and colleagues (2021) assessed absolute agreement in ratings.”
BackgroundMethodResultInfluentialSelf-citationNudging Social Media toward Accuracy
Pennycook, G., & Rand, D. G. (2022) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/00027162221092342 · Cited by 106
“…accuracy), they are often very good at distinguishing between true and false news headlines (e.g., averaging the accuracy ratings of even small groups of laypeople can produce as much agreement with professional fact-checkers as professional fact-checkers show with each other; Allen et al. 2021).”
BackgroundSelf-citationMisinformation exploits outrage to spread online
McLoughlin, K. L., Brady, W. J., Goolsbee, A., Kaiser, B., Klonick, K., & Crockett, M. J. (2024) · Science · doi:10.1126/science.adl2829 · Cited by 106
The state of human-centered NLP technology for fact-checking
Das, A., Liu, H., Kovatchev, V., & Lease, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103219 · Cited by 89
True or False: Studying the Work Practices of Professional Fact-Checkers
Micallef, N., Armacost, V., Memon, N., & Patil, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512974 · Cited by 70
“Fact-checking is one of the most efective mechanisms to counter misinformation [4].”
BackgroundMethodInfluentialCommunity-Based Fact-Checking on Twitter’s Birdwatch Platform
Pröllochs, N. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19335 · Cited by 86
High level of correspondence across different news domain quality rating sets
Lin, H., Lasser, J., Lewandowsky, S., Cole, R., Gully, A., Rand, D. G., & Pennycook, G. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad286 · Cited by 94
“A large body of research has focused on understanding why people consume or share misinformation (1 – 4), and many interventions have been developed to combat the spread of misinformation online (5 – 7).”
Self-citationModerating with the Mob: Evaluating the Efficacy of Real-Time Crowdsourced Fact-Checking
Godel, W., Sanderson, Z., Aslett, K., Nagler, J., Bonneau, R., Persily, N., & Tucker, J. A. (2021) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i1.15 · Cited by 66
“We therefore complement the findings of Allen et al. (2021) by testing the efficacy of crowdsourced fact-checking in real time using a transparent, preregistered method for sampling popular articles.”
BackgroundMethodResultInfluentialCan Online Juries Make Consistent, Repeatable Decisions?
Hu, X. E., Whiting, M. E., & Bernstein, M. S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445433 · Cited by 62
“Related research has found, for example, that deliberation reduces the belief in false news headlines [8] and that the wisdom of crowds can enable laypeople to predict professional fact-checkers’ opinions [3].”
BackgroundDo Explanations Increase the Effectiveness of AI-Crowd Generated Fake News Warnings?
Epstein, Z., Foppiani, N., Hilgard, S., Sharma, S., Glassman, E. L., & Rand, D. G. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19283 · Cited by 66
Self-citationCommunity notes increase trust in fact-checking on social media
Drolsbach, C., Solovev, K., & Pröllochs, N. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae217 · Cited by 82
“The assessment of even relatively small crowds is comparable to those of experts (33, 34, 44, 45, 47, 48, 68).”
Leveraging ChatGPT for Efficient Fact-CheckingPreprint
Hoes, E., Altay, S., & Bermeo, J. (2023) · doi:10.31234/osf.io/qnjkf · Cited by 61
Displaying News Source Trustworthiness Ratings Reduces Sharing Intentions for False News Posts
Celadin, T., Capraro, V., Pennycook, G., & Rand, D. G. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i5.100 · Cited by 52
Self-citationSpotting false news and doubting true news: a systematic review and meta-analysis of news judgements
Pfänder, J., & Altay, S. (2025) · Nature Human Behaviour · doi:10.1038/s41562-024-02086-1 · Cited by 89
“In line with past work 38 , we have shown that this was not only true on average, but also for a large majority (79.92”
InfluentialAustin, E. W., Borah, P., & Domgaard, S. (2021) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-58 · Cited by 50
He, B., Ahamad, M., & Kumar, S. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583388 · Cited by 51
“They complement fact checkers who can only verify a handful of stories after they have gone viral [3, 42].”
BackgroundLateral reading and monetary incentives to spot disinformation about science
Panizza, F., Ronzani, P., Martini, C. M. C., Mattavelli, S., Morisseau, T., & Motterlini, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-09168-y · Cited by 47
Changing the incentive structure of social media platforms to halt the spread of misinformation
Globig, L. K., Holtz, N., & Sharot, T. (2023) · eLife · doi:10.7554/elife.85767 · Cited by 56
Chen, Y.-P., Chen, Y., Yang, K.-C., Lai, F., Huang, C., Chen, Y.-N., & Tu, Y.-C. (2022) · Journal of Medical Internet Research · doi:10.2196/36830 · Cited by 41
“Experts or the wisdom of crowds can detect false information manually [27,34], but efficiency can be an issue when news may have spread before a judgment was made.”
Differences in misinformation sharing can lead to politically asymmetric sanctions
Mosleh, M., Yang, Q., Zaman, T. R., Pennycook, G., & Rand, D. G. (2024) · Nature · doi:10.1038/s41586-024-07942-8 · Cited by 35
Self-citationHow Effective Are Interventions Against Misinformation?
Altay, S. (2022) · doi:10.31234/osf.io/sm3vk · Cited by 32
Lesher, M., Pawelec, H., & Desai, A. (2022) · OECD going digital toolkit notes · doi:10.1787/84b62df1-en · Cited by 31
In generative AI we trust: can chatbots effectively verify political information?
Kuznetsova, E., Makhortykh, M., Vziatysheva, V., Stolze, M., Baghumyan, A., & Urman, A. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00338-8 · Cited by 32
Fact-checking in the age of AI: Reducing biases with non-human information sources
Moon, W.-K., & Kahlor, L. A. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102760 · Cited by 34
van der Linden, S., Albarracín, D., Fazio, L. K., Freelon, D., Roozenbeek, J., Swire‐Thompson, B., & Bavel, J. J. V. (2023) · PsycEXTRA Dataset · doi:10.1037/e506432023-001 · Cited by 32
Understanding and Combating Misinformation Across 16 Countries on Six Continents
Arechar, A. A., Allen, J. N. L., Berinsky, A. J., Cole, R., Epstein, Z., Garimella, K., Gully, A., Lu, J. G., Ross, R. M., Stagnaro, M., et al. (2022) · doi:10.31234/osf.io/a9frz · Cited by 25
Self-citationCommunity Notes vs. Snoping: How the Crowd Selects Fact-Checking Targets on Social Media
Pilarski, M., Solovev, K., & Pröllochs, N. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31387 · Cited by 35
Ma, H. (., Huang, W. W., & Dennis, A. R. (2024) · Journal of Management Information Systems · doi:10.1080/07421222.2024.2376381 · Cited by 28
Crowdsourced Fact-checking: Does It Actually Work?
Barbera, D. L., Maddalena, E., Soprano, M., Roitero, K., Demartini, G., Ceolin, D., Spina, D., & Mizzaro, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103792 · Cited by 25
On the Efficacy of Accuracy Prompts Across Partisan Lines: An Adversarial Collaboration
Martel, C., Rathje, S., Clark, C. J., Pennycook, G., Bavel, J. J. V., Rand, D. G., & van der Linden, S. (2024) · Psychological Science · doi:10.1177/09567976241232905 · Cited by 34
“For example, even when nudged toward accuracy, Republicans may be worse than Democrats at identifying true versus false headlines (e.g., J. Allen et al., 2021; Garrett & Bond, 2021; Imhoff et al., 2022; Lawson & Kakkar, 2022; Rathje, Van Bavel, & van der Linden, 2023; van der Linden, Panagopoulos,…”
BackgroundSelf-citationMahl, D., Zeng, J., Schäfer, M. S., Egert, F. A., & de Oliveira, T. M. (2024) · The International Journal of Press/Politics · doi:10.1177/19401612241270004 · Cited by 35
“…2007) to tease out the specificities of each, as well as exploring whether and how the hijacking of professional fact-checking practices by malicious actors, the weaponization of fact-checking, or crowdsourced fact-checks (Allen et al. 2021) might push the boundaries of fact-checking cultures.”
Community notes reduce engagement with and diffusion of false information online
Slaughter, I., Peytavin, A., Ugander, J. H., & Saveski, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2503413122 · Cited by 51
“This structural difference contrasts notably with differences between false vs. true content diffusion itself, where false information diffuses farther, but with structural patterns that are otherwise indistinguishable from those of true information, conditional on reach. misinformation |”
Can Crowdsourcing Rescue the Social Marketplace of Ideas?
Yasseri, T., & Menczer, F. (2023) · Communications of the ACM · doi:10.1145/3578645 · Cited by 22
Neumann, T., De‐Arteaga, M., & Fazelpour, S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533205 · Cited by 29
“To do so, it uses a corpus of previously fact-checked claims stemming from either crowd-sourced knowledge, such as Wikipedia [70] or surveys [2], or from professional assessments, such as Snopes or PolitiFact [66].”
BackgroundDelphiAgent: A trustworthy multi-agent verification framework for automated fact verification
Xiong, C. C., Zheng, G., Ma, X., Li, C., & Zeng, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104241 · Cited by 23
Untangling the web of misinformation and false beliefs
Johar, G. V. (2022) · Journal of Consumer Psychology · doi:10.1002/jcpy.1288 · Cited by 21
High level of correspondence across different news domain quality rating setsPreprint
Lin, H., Lasser, J., Lewandowsky, S., Cole, R., Gully, A., Rand, D. G., & Pennycook, G. (2022) · doi:10.31234/osf.io/qy94s · Cited by 20
Self-citationBerger, L., Kerkhof, A., Mindl, F., & Münster, J. (2025) · Journal of Public Economics · doi:10.1016/j.jpubeco.2025.105345 · Cited by 38
Markowitz, D. M., Levine, T. R., Serota, K. B., & Moore, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289004 · Cited by 27
“A crucial question is: How do we know the factcheckers are accurate? The answer is to check and compare the fact-checkers [6, 7].”
BackgroundLisnic, M., Lex, A., & Kogan, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642448 · Cited by 29
Johansson, P., Enoch, F., Hale, S. A., Vidgen, B., Bereskin, C., Margetts, H. Z., & Bright, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4648332 · Cited by 30
Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting
Blanchard, M. D., Herzog, S. M., Kämmer, J. E., Zöller, N., Kostopoulou, O., & Kurvers, R. H. J. M. (2024) · Medical Decision Making · doi:10.1177/0272989x241241001 · Cited by 15
“30 The pooling of independent decisions has been successfully applied to a diverse range of tasks, including the prediction of election outcomes, 31 memory retrieval, 32 fingerprint analysis, 33 false news identification, 34 and medical decision making.”
MethodDiverse misinformation: impacts of human biases on detection of deepfakes on networks
Lovato, J., St-Onge, J., Harp, R., Salazar Lopez, G., Rogers, S. P., Haq, I. U., Hébert-Dufresne, L., & Onaolapo, J. (2024) · npj Complexity · doi:10.1038/s44260-024-00006-y · Cited by 16
“Non-dupedneighborscanthencorrecttheirdupedneighborsatarate γ 38 – 40 ,e.g.,we assume that your network neighbors can fact-check something you diffuse online and potentially correct your opinion.”
InfluentialAkhtar, M. M., Masood, R., Ikram, M., & Kanhere, S. S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3644998 · Cited by 16
“Such collective intelligence allows for scalable fact-checking that complements the work of professional fact-checkers, as evidenced by recent research [3].”
BackgroundSincere or Motivated? Partisan Bias in Advice-taking
Zhang, Y., & Rand, D. G. (2021) · Judgment and Decision Making · doi:10.1017/jdm.2023.28 · Cited by 16
Self-citation'Debunk-It-Yourself': Health Professionals Strategies for Responding to Misinformation on TikTok
Sharevski, F., Loop, J. V., Jachim, P., Devine, A., & Das, S. (2024) · New Security Paradigms Workshop · doi:10.1145/3703465.3703469 · Cited by 15
“…the “wisdom of the crowds” and applying community notes instead of labels, though promising, have the disadvantage of disputes among users, which, evidence shows, find it hard to leave their preexisting political, ideological, or other biases aside when debunking claims on social media [1].”
Believability and Harmfulness Shape the Virality of Misleading Social Media Posts
Drolsbach, C., & Pröllochs, N. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583857 · Cited by 22
“Specifcally, the idea is to let regular social media users carry out fact-checking of social media posts [2, 3, 10, 15, 20, 29, 38].”
BackgroundResnick, P., Alfayez, A., Im, J., & Gilbert, É. (2023) · Collective Intelligence · doi:10.1177/26339137231173407 · Cited by 14
“The results of this study, together with that of (Allen et al., 2021), demonstrate a quality-cost tradeoff.”
BackgroundMethodResultInfluentialShin, J., Lewis, S. C., Kim, S., & Thorson, K. (2024) · New Media & Society · doi:10.1177/14614448241228851 · Cited by 14
“As for accuracy assessment, recent findings (Allen et al., 2021; Pennycook and Rand, 2019; Resnick et al., 2023) showed a high level of consistency between journalists and regular audiences.”
InfluentialHe, B., Hu, Y., Lee, Y.-C., Oh, S., Verma, G., & Kumar, S. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3694980 · Cited by 14
“On the other hand, professional fact-checkers face constraints in terms of the limited number of fact-checkers and the significant time required for label generation; moreover, their fact-checks tend to address only a small number of viral claims [4, 58, 70].”
InfluentialCorsi, G., Seger, E., & hÉigeartaigh, S. Ó. (2024) · Online Social Networks and Media · doi:10.1016/j.osnem.2024.100291 · Cited by 14
Wirtschafter, V., & Majumder, S. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v2i1.139 · Cited by 18
“The use of volunteers—culled from a platform’s community of users—is also becoming an increasingly prominent approach, particularly as social media companies seek to cut costs, increase revenue, and deflect responsibility for potentially controversial moderation decisions (Allen et al. 2021).”
BackgroundReferences to unbiased sources increase the helpfulness of community fact-checks
Solovev, K., & Pröllochs, N. (2025) · Scientific Reports · doi:10.1038/s41598-025-09372-6 · Cited by 19
“A community-based approach to fact-checking is promising as it offers the capacity to conduct numerous fact-checks at high frequency and low costs [14, 17].”
InfluentialCoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation
Venkatagiri, S., Mukhopadhyay, A., Hicks, D. L., Brantly, A. F., & Luther, K. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3595997 · Cited by 12
“[5] found that crowdsourced trustworthiness ratings can distinguish between real and fake news sources,”
BackgroundHow do online users respond to crowdsourced fact-checking?
Panizza, F., Ronzani, P., Morisseau, T., Mattavelli, S., & Martini, C. M. C. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02329-y · Cited by 13
Conversational Agents to Facilitate Deliberation on Harmful Content in WhatsApp Groups
Agarwal, D., Shahid, F., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687030 · Cited by 19
“Moreover, centralized fact-checking faces scalability problems as there is far more content online than is possible for human fact-checkers to assess, especially when fact-checking cannot be fully automated [3, 35].”
BackgroundCombining Human and Machine Confidence in Truthfulness Assessment
Qu, Y., Roitero, K., Barbera, D. L., Spina, D., Mizzaro, S., & Demartini, G. (2023) · Journal of Data and Information Quality · doi:10.1145/3546916 · Cited by 6
“In order to best deal with the need to both scale to large number of fact-checks and have expert journalists manually checking and evaluating the veracity of posted information, human-in-the-loop systems have been considered [1, 8, 25].”
BackgroundThe Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk Evaluation
Duan, X., Ho, C.-J., & Yin, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512239 · Cited by 12
“For example, the crowd is asked to rate the credibility of news content [2], to assess the performance of their peers in massive open online courses [15], and to make judgements about other people”
BackgroundCollaboratively adding context to social media posts reduces the sharing of false news.Preprint
Renault, T., Amariles, D. R., & Troussel, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4800565 · Cited by 29
“The results from this literature are mixed: while labels from expert fact-checkers tend to be more universally effective and align with established content moderation practices, labels generated through community-sourced fact-checking have also shown effectiveness (Allen et al., 2021).”
BackgroundCombining Large Language Models and Crowdsourcing for Hybrid Human-AI Misinformation Detection
Zeng, X., Barbera, D. L., Roitero, K., Zubiaga, A., & Mizzaro, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657965 · Cited by 17
“Allen et al. [1] found correlation between politically balanced crowds and experts on news headlines, while Saeed et al. [21] found similar patterns on Twitter data.”
BackgroundNudging social media sharing towards accuracyPreprint
Pennycook, G., & Rand, D. G. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/tp6vy · Cited by 12
Self-citationConsidering information-sharing motives to reduce misinformation
Globig, L. K., & Sharot, T. (2024) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2024.101852 · Cited by 10
Measuring receptivity to misinformation at scale on a social media platform
Tokita, C. K., Aslett, K., Godel, W., Sanderson, Z., Tucker, J. A., Nagler, J., Persily, N., & Bonneau, R. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae396 · Cited by 14
“The largest gain in effectiveness could come from methods that speed up the article verification process, offering faster turn around than traditional fact-checking — methods such as machine learning (63, 64) or crowd-sourcing (63, 65, 66).”
BackgroundMatamoros-Fernández, A., & Jude, N. (2025) · New Media & Society · doi:10.1177/14614448251314399 · Cited by 16
“…been criticised for their limited ability to compute context, nuance and problematic meta-narratives (Griffin, 2023), much faith has been placed in their capabilities when combined with volunteer moderators, their potential otherwise referred to as ‘the wisdom of the crowd’ (Allen et al., 2021).”
When More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21994 · Cited by 12
Miyazaki, K., Uchiba, T., Tanaka, K., An, J., Kwak, H., & Sasahara, K. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22176 · Cited by 12
Interactive crowdsourcing to fact-check politicians.
Espina-Mairal, S., Bustos, F., Solovey, G., & Navajas, J. (2023) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000492 · Cited by 13
Community notes reduce the spread of misleading posts on XPreprint
Chuai, Y., Pilarski, M., Lenzini, G., & Pröllochs, N. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3a4fe · Cited by 13
(2021) · doi:10.31234/osf.io/ar96c · Cited by 12
Self-citationHasan, E., Duhaime, E., & Trueblood, J. S. (2024) · Cognitive Research Principles and Implications · doi:10.1186/s41235-024-00558-6 · Cited by 8
“…the wisdom of the crowd approaches has emerged as a powerful approach to solving many complicated classification problems including misinformation (Allen et al., 2021) and deep fake detection (Groh et al., 2022) as well as medical image decision-making (Kurvers et al., 2016; Hasan et al., 2023;…”
BackgroundSpotting False News and Doubting True News: A Meta-Analysis of News JudgmentsPreprint
Pfänder, J., & Altay, S. (2023) · doi:10.31219/osf.io/n9h4y · Cited by 9
Stewart, A. J., Arechar, A. A., Rand, D. G., & Plotkin, J. B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2315195121 · Cited by 16
“We also assessed each story’s perceived accuracy by recruiting 1, 000 American participants from Amazon Mechanical Turk to rate the accuracy of 20 headlines (yielding a total of 20, 000 accuracy ratings), which has been shown to produce good agreement with the ratings of professional fact-checkers via the wisdom of cr…”
MethodSelf-citationKim, J., Wang, Z., Shi, H., Ling, H.-K., & Evans, J. A. (2025) · Nature Communications · doi:10.1038/s41467-025-55868-0 · Cited by 8
“Literature on the wisdom of crowds suggests that while individual tags are susceptible to biases and noise, aggregating tags collectively could correct individual bias, increasing the quality of nonexpert fact-checks 28,52,53 .”
Abels, C. M., Lopez-Lopez, E., Burton, J. W., Holford, D. L., Brinkmann, L. U. L., Herzog, S. M., & Lewandowsky, S. (2025) · Behavioral Science & Policy · doi:10.1177/23794607251347020 · Cited by 14
“Such crowdsourced reporting 41,42 could provide early warnings for emerging issues that regulators might have missed.”
Jornalistas Locais e Fact-Checking: Um Estudo Exploratório em Portugal e Espanha
Jerónimo, P., & Esparza, M. S. (2023) · Comunicação e Sociedade · doi:10.17231/comsoc.44(2023).4553 · Cited by 8
The unappreciated role of intent in algorithmic moderation of abusive content on social media
Wang, X., Koneru, S., Venkit, P. N., Frischmann, B. M., & Rajtmajer, S. M. (2025) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-180 · Cited by 14
Emotional language reduces belief in false claims
Phillips, S. C., Wang, S. Y. N., Carley, K. M., Rand, D. G., & Pennycook, G. (2025) · Judgment and Decision Making · doi:10.1017/jdm.2025.10019 · Cited by 11
“Crowd-sourced fact-checking has drawn particular attention due to its scalability (Allen et al., 2021; Martel and Rand, 2023).”
Self-citationCommunity-based fact-checking reduces the spread of misleading posts on X (formerly Twitter)
Chuai, Y., Pilarski, M., Renault, T., Amariles, D. R., Troussel, A., Lenzini, G., & Pröllochs, N. (2026) · Nature Communications · doi:10.1038/s41467-026-72597-0 · Cited by 11
Realtime user ratings as a strategy for combatting misinformation: an experimental study
Stein, J., Frey, V., & van de Rijt, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-28597-x · Cited by 8
“We further compare these scenarios with a control condition resembling the setup in which crowd-based ratings have been studied previously 7,9 : namely, a scenario in which subjects rate messages independently and without information about the rating decisions of others.”
BackgroundResultInfluentialYaqub, W., Kay, J., & Goldwater, M. B. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637335 · Cited by 9
“Literature described above on misinformation has been dominated by large-scale quantitative studies and on American crowd-source samples [3, 4, 18, 24, 26, 41, 47, 49, 66, 72–74, 76, 89].”
BackgroundUnbiased misinformation policies sanction conservatives more than liberalsPreprint
Mosleh, M., Yang, Q., Zaman, T. R., Pennycook, G., & Rand, D. G. (2022) · doi:10.31234/osf.io/ay9q5 · Cited by 7
Self-citationNewsComp: Facilitating Diverse News Reading through Comparative Annotation
Bhuiyan, M. M., Lee, S. W., Goyal, N., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581244 · Cited by 7
“On the other hand, studies have shown that crowdworkers’ output can be signifcantly correlated with experts’ in some annotation tasks [2, 3, 9, 11, 74].”
BackgroundYours Truly: A Credibility Framework for Effortless LLM-Powered Fact Checking
Krishnamurthy, V., & Balaji, V. (2024) · IEEE Access · doi:10.1109/access.2024.3520187 · Cited by 15
“Misleading can refer to a combination of true & false claims [32].”
Limited effectiveness of psychological inoculation against misinformation in a social media feed
Wang, S.-Y. N., Phillips, S. C., Carley, K. M., Lin, H., & Pennycook, G. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf172 · Cited by 10
Self-citation
Metadata
- Authors
- Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Nottingham (GB)
- University of Regina (CA)
- Corresponding author
- Allen, Jennifer
- Arechar, Antonio A.
- Rand, David G.
- Venue
- Science Advances
- Published
- 2021-09-03
- Volume
- 7
- Issue
- 36
- Pages
- eabf4393-eabf4393
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC8442902
- OpenAlex
- W3172233798
- Cited by
- 234
- References
- 56
- Citations by year
- 2026: 44 · 2025: 47 · 2024: 48 · 2023: 45 · 2022: 33 · 2021: 15
- FWCI
- 49.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Science Advances (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- PsyArXiv (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- John Templeton Foundation (61061)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- fact-checking, misinformation, crowdsourcing, wisdom of crowds, laypeople, political balance, Facebook, news headlines, cognitive reflection, political knowledge, partisanship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
