The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 13 of the 41 sources its claims cite were checked for support.
Deepfake detection by human crowds, machines, and machine-informed crowds
Open accessAbstract
Significance: The recent emergence of deepfake videos raises theoretical and practical questions. Are humans or the leading machine learning model more capable of detecting algorithmic visual manipulations of videos? How should content moderation systems be designed to detect and flag video-based misinformation? We present data showing that ordinary humans perform in the range of the leading machine learning model on a large set of minimal context videos. While we find that a system integrating human and model predictions is more accurate than either humans or the model alone, we show inaccurate model predictions often lead humans to incorrectly update their responses. Finally, we demonstrate that specialized face processing and the ability to consider context may specially equip humans for deepfake detection.
Overview
CoChat Summary
TL;DR
In two online experiments, ordinary people performed near or above the leading 2020 deepfake-detection model on minimal-context DFDC videos, and crowd averages matched or exceeded the model. Combining human and model predictions improved accuracy overall, but incorrect model predictions often led participants to update toward wrong answers.
Key findings
- Experiment 1: 5,524 individuals completed 26,820 trials; among 882 participants who saw at least 10 pairs, 82% outperformed the leading model, which achieved 65% accuracy on the holdout dataset. Experiment 2: recruited participants accurately identified deepfakes in 66% of attempts, nonrecruited participants in 69% of trials (72% for those seeing at least 10 videos), versus the leading model’s 80% on sampled videos; 13% of recruited, 27% of nonrecruited with ≥10 videos, and 37% of nonrecruited with <10 videos outperformed the model.
- Crowd wisdom: the crowd mean accurately identified 74% of videos for recruited participants, 80% for nonrecruited participants, and 86% for the 1,879 nonrecruited participants who saw at least 10 videos, compared with 80% for the leading model. After seeing the model’s prediction, recruited participants’ accurate identification increased from 66% to 73% (P < 0.001), but on 10 videos with incorrect or equivocal model predictions, participants became 2.7% less accurate on average; on political deepfakes the model was confidently wrong (2%, 8%, and 1% deepfake probability) and participants’ accurate identification dropped from 56% to 34% for Kim Jong-un and 70% to 55% for Vladimir Putin.
- Specialized face processing: inversion, misalignment, and eye occlusion reduced human accuracy by 4.3 percentage points (P = 0.002), 6.3 percentage points (P < 0.001), and 4.4 percentage points (P = 0.004) on a base rate of 65.5%; the model was affected only by inversion, dropping 12.1 percentage points (P = 0.005). Anger elicitation had no overall significant effect (P = 0.280), but on real videos the anger group underperformed controls by 5.2 percentage points (P = 0.032).
- Video features: a flickering face increased recruited participants’ accuracy by 24.2 percentage points (P < 0.001) and the model’s by 21.7 percentage points (P = 0.170); two people increased recruited participants’ accuracy by 7.6% (P < 0.001) but decreased the model’s accuracy by 21.9% (P = 0.023) on real videos; dark-skin presence decreased recruited participants’ accuracy by 8.8% (P < 0.001) and the model’s by 12.0% (P = 0.192).
Methods
- Two randomized online experiments on the Detect Fakes website. Experiment 1 used a two-alternative forced-choice design with 56 pairs of videos from the DFDC training dataset (n = 5,524; 26,820 trials; analysis focused on 882 participants who saw at least 10 pairs). Experiment 2 used a single-video design (n = 9,492: 304 recruited from Prolific and 9,188 organic visitors) with 50 videos randomly sampled from the DFDC holdout set (half deepfake, half nonmanipulated), 4 political-leader videos, and a deepfake attention check; participants reported confidence from 50% to 100% in 1% increments and could update after seeing the model’s prediction.
- Embedded randomized interventions tested anger elicitation and obstruction of specialized face processing via inversion, misalignment, and occlusion. Analyses used linear regression with robust SEs clustered on participants unless otherwise stated; accuracy was normalized between 0 and 1, with crowd means, ROC/AUC comparisons, and hand-labeled video-level features.
Limitations
- Paper states: the study evaluated 167 videos (84 deepfake and 83 authentic); only the two political deepfake videos include lip-syncing manipulations; no expert fact-checkers or forensic analysts were recruited, so results generalize only to ordinary people; the comparison is limited to the best model in 2020; minimal-context videos may not resemble the most problematic deepfakes; the half-real/half-deepfake base rate does not reflect real misinformation base rates; and emotion results were mixed, so no firm conclusions are drawn.
- Paper states: future work is needed on persuasive manipulated video, larger samples of political deepfakes, social media context, and additional or integral emotions.
- Evident from text: the nonrecruited participants were not preregistered and arrived organically/self-selected via the public website, while recruited participants came from Prolific.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
5 Valid · 1 Not valid · 18 Not checked · 170 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“adults are less accurate at recognizing faces when images are inverted or contain misaligned parts (60–62).”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Abstract
What the sources say
- Not checked yetcites [60] R. K. Yin, Looking at upside-down faces. J. Exp. Psychol. 81, 141– 145 (1969). source
- Not checked yetcites [61] G. Rhodes, S. Brake, A. P. Atkinson, What’s lost in inverted faces? Cognition 47, 25– 57 (1993). 8482070 source
- Disputedcites [62] J. J. Richler, O. S. Cheung, I. Gauthier, Holistic processing predicts face recognition. Psychol. Sci. 22, 464– 471 (2011). 21393576 PMC3077885 sourceExperimental — accuracy not yet measured
Our copy never reports an inversion cost for face recognition — 'inverted' appears only inside reference titles (Richler, Mack, Palmeri & Gauthier, 2011; Goffaux, 2009; Duchaine & Nakayama, 2006), which is not the paper's own claim. On alignment, the paper states the opposite direction: in the Method > Composite task section it writes that in the partial design 'Accuracy is greater or reaction time (RT) is faster in misaligned trials than in aligned trials,' and in Results and discussion that 'when face parts are misaligned, selective attention to a part is more successful, as evidenced by the smaller congruency effect on misaligned trials.' Its face-recognition measures (CFMT, face-identification task) used intact upright faces varying in lighting and pose, with no misaligned-parts or inverted condition, so the attributed decrement is not what this source says. (LLM, conf=0.65)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“visual processing of faces appears to proceed holistically for the vast majority of people (64, 65).”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Abstract
What the sources say
- Partly supportedcites [64] J. J. Richler, I. Gauthier, A meta-analysis and review of holistic face processing. Psychol. Bull. 140, 1281– 1302 (2014). 24956123 PMC4152424 sourceExperimental — accuracy not yet measured
The Holistic Processing section describes face processing as holistic, and the Meta-Analysis Results report an average holistic-processing effect across studies. However, those group-level findings do not establish that the effect occurs in the vast majority of individual people. (LLM, conf=0.82)
- Supportedcites [65] A. W. Young, A. M. Burton, Are we face experts? Trends Cogn. Sci. 22, 100– 110 (2018). 29254899 source
In “The facial expertise debate” (page not provided), the paper calls holistic processing a well-known property of face recognition. Its “Holistic processing” entry defines this as processing faces as a whole. Although it gives no percentage, these passages support the citing sentence’s broad, hedged description of typical face processing. (LLM, conf=0.83)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Likewise, these results mirror other recent research revealing human–AI collaborative decision-making does not necessarily lead to more accurate results than either humans or AI alone (72–76).”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Discussion
What the sources say
- Not checked yetcites [72] M. Vaccaro, J. Waldo, The effects of mixing machine learning and human judgment. Commun. ACM 62, 104– 110 (2019). source
- Not checked yetcites [73] P. Tschandl., Human-computer collaboration for skin cancer recognition. Nat. Med. 26, 1229– 1234 (2020). 32572267 source
- Partly supportedcites [74] S. Gaube., Do as AI say: Susceptibility in deployment of clinical decision-aids. NPJ Digit. Med. 4, 31 (2021). 33608629 PMC7896064 sourceExperimental — accuracy not yet measured
Our copy supports the 'collaboration does not necessarily yield better accuracy' half: the Results section 'Diagnostic accuracy is similarly impacted by human/AI advice' reports accuracy was worse with inaccurate advice regardless of purported source, and the Discussion states 'physicians were not able to effectively filter inaccurate advice'. However, the comparison 'than either humans or AI alone' is not tested here — the Methods 'Experimental design' describes a 2 × 2 design varying only advice source (AI vs human label) and advice accuracy, with no unaided-physician or AI-alone condition, and the Discussion instead says outcomes 'may improve overall' only 'if the underlying model has a higher diagnostic accuracy than the physicians using it'. The related statement that implemented systems 'showed no improved clinical outcomes' (Introduction) is itself credited to other works (refs 6,7) rather than shown by this study. (LLM, conf=0.78)
- Partly supportedcites [75] A. Abeliuk, D. M. Benjamin, F. Morstatter, A. Galstyan, Quantifying machine influence over human forecasters. Sci. Rep. 10, 15940 (2020). 32994447 PMC7524768 sourceExperimental — accuracy not yet measured
The Discussion (para. on implications) states that 'algorithms are far from perfect and do not necessarily beat humans in some deeply uncertain settings' and that 'overconfidence on algorithms may lead to detrimental results', and the Impact on performance / Discussion sections report that confirmation bias and trust in low-quality models decrease forecast accuracy ('users would do better to trust their prior'), so the source does support the idea that pairing humans with machine models does not necessarily beat humans alone. However, the paper never compares the hybrid against the machine/AI component alone; its own headline conclusion is the more positive 'machine models can improve the judgment of a human pool but highlight the importance of accounting for trust and cognitive biases' (Abstract), and the Machine predictions section frames machines as under-performing humans rather than the team being worse than machines. The 'or AI alone' half of the claim is therefore not supported by this source. (LLM, conf=0.75)
- Supportedcites [76] M. Jacobs., How machine-learning recommendations influence clinician treatment selections: The example of the antidepressant selection. Transl. Psychiatry 11, 108 (2021). 33542191 PMC7862671 source
The Results section 'No change in overall accuracy between clinicians and clinician-ML collaboration' (p. Results) reports clinician-ML collaborative accuracy (M = 0.356) statistically indistinguishable from clinicians alone (M = 0.357; p = 0.925) and significantly below the simulated ML system acting alone (0.667; p < 0.0001). The Discussion restates this: 'interacting with ML recommendations did not improve treatment selection accuracy' and 'Our work challenges the validity of this assumption' that humans interacting with ML tools will perform better than either actor individually. This matches the cited sentence's paraphrase that human-AI collaboration does not necessarily yield more accurate results than either humans or AI alone. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Eighty-two percent of participants outperform the leading model, which achieves 65% accuracy on the holdout dataset (36).”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Results > Experiment 1: Two-Alternative Forced Choice (n = 5,524).
What the sources say
- Not checked yetcites [36] B. Dolhansky., Deepfake detection challenge results: An open initiative to advance AI (2020). Accessed 27 January 2021.
plainn = 882- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Half of the stimuli set (28 of 56 pairs of videos) was identified correctly by over 83% of participants”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Results > Experiment 1: Two-Alternative Forced Choice (n = 5,524).
plainn = 882- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
126 more claims
Create a free account to see all 194 claims and to verify the 28 references still open.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 31
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no improvement when evaluating videos either [24–26].”
From Warning: Humans cannot reliably detect speech deepfakes (2023) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On a sample of 166 videos from the largest publicly available dataset of deepfake videos to date from the Deepfake Detection Contest (DFDC)48, people are significantly better than chance but far from perfect at discerning whether an unknown actor’s face has been visually manipulated by a deepfake algorithm49.”
From Human detection of political speech deepfakes across transcripts, audio, and video (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, Groh et al. [2] point out that the brain possesses a specialized region for processing faces holistically. In their research, humans were adversely a ff ected by manipulations obstructing specialized face processing (e.g., inversion, misalignment, and eye occlusion), whereas computer performance remained una ff ected.”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Groh et al. [2] compared human performance with the winning model using videos from DFDC and found that, among the participants who saw at least 10 videos, 82% outperformed the winning model.”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, while lower-quality deepfakes are generally easier to detect [20, 35, 36, 42], imperfections in authentic videos (e.g., lighting inconsistencies or unusual framing) can lead viewers to miscategorize genuine content as manipulated [2, 22, 27, 35, 37].”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measured
12 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Results > Experiment 2: Single-Video Design (n = 9,492).
“For nonrecruited participants, the crowd mean accurately identifies 80% of videos.”
§ Deepfake detection by human crowds, machines, and machine-informed crowds > Results > Experiment 2: Single-Video Design (n = 9,492).
“the crowd wisdom of nonrecruited participants and the model is correct on 10 videos. When we examine the 36 videos that are neither grainy, blurry, nor extremely dark, the crowd wisdom of recruited participants is correct on 29 out of 36 videos, the crowd wisdom of nonrecruited participants is correct on 32 out of 36 videos”
Citations
Cites
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 2×
“While there are many proposed techniques for algorithmically detecting fakes (including affective computing approaches like examining heart rate and breathing rate (37) and looking for emotion-congruent speech and facial expressions) (38, 39), the most accurate computer vision model in the contest (40) focused on loca…”
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“The role of emotion in deepfake detection is of practical concern because people share misinformation, especially political misinformation, because of its novelty and emotional content (55).”
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738 · cited 1×
“For example, there is a region of the brain specialized for processing faces (57).”
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
(2018) · Cited by 3,510
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
Deep learning-aided decision support for diagnosis of skin disease across skin tones
Groh, M., Badri, O., Daneshjou, R., Koochek, A., Harris, C., Soenksen, L. R., Doraiswamy, P. M., & Picard, R. (2024) · Nature Medicine · doi:10.1038/s41591-023-02728-3 · Cited by 160
Self-citationAlgorithm-mediated social learning in online social networks
Brady, W. J., Jackson, J. C., Lindström, B., & Crockett, M. J. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.06.008 · Cited by 132
Abbas, F., & Taeihagh, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124260 · Cited by 184
Danry, V., Pataranutaporn, P., Mao, Y., & Maes, P. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580672 · Cited by 110
“a member of a minority group which can lead to over-policing of marginalized communities, or a social media information recommendation system might falsely claim incorrect or misleading news stories to be true which can lead to political instability and social unrest [17, 18, 46].”
Background
Metadata
- Authors
- Groh, M., Epstein, Z., Firestone, C., & Picard, R.
- Affiliations
- MIT Media Lab (US)
- Johns Hopkins University (US)
- Corresponding author
- Groh, Matthew
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-12-28
- Volume
- 119
- Issue
- 1
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8740705
- OpenAlex
- W3209972423
- Cited by
- 248
- References
- 100
- Citations by year
- 2026: 64 · 2025: 55 · 2024: 65 · 2023: 43 · 2022: 20 · 2021: 1
- FWCI
- 11.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- arXiv (Cornell University)
- Funders
- MIT Media Lab member companies (NA)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- deepfake detection, human-machine comparison, face processing
- MeSH terms
- Facial Recognition, Artificial Intelligence, Communication, Humans, Deception, Video Recording, Forensic Sciences, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deepfake detection by human crowds, machines, and machine-informed crowds. https://app.cochat.ai/papers/10.1073/pnas.2110013119
- The paper
- Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022). Deepfake detection by human crowds, machines, and machine-informed crowds. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2110013119
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.