Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 100 of the 101 references it cites were found in a registry or an index; 20 of the 75 sources its claims cite were checked for support.

Human detection of political speech deepfakes across transcripts, audio, and video

Open access

Abstract

Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.

Overview

CoChat Summary

## TL;DR

This paper runs 5 pre-registered randomized experiments with 2,215 recruited participants to test how well humans detect fabricated political speeches across text, audio, and video. Overall, people are more accurate when audio and visual cues are available than when judging transcripts alone, and the paper finds no statistically significant effect of misinformation base rate on overall discernment.

## Key findings

  • In Experiment 1a (N = 501), accuracy rose with more modalities: 57% for transcripts, 64% for silent video without subtitles, 69% for silent video with subtitles, 81% for audio, 85% for video with audio and subtitles, and 86% for video with audio but no subtitles.
  • In Experiment 1b (41,313 participants; not preregistered), the same ordering held, from 43% accuracy for transcripts up to 83% for video with audio but no subtitles.
  • In Experiment 3 (N = 1,006), the base rate of fakes did not significantly change overall accuracy, but a high-fake condition increased accuracy on real stimuli by 7.2 percentage points and decreased accuracy on fabricated stimuli by 5.8 percentage points; video with audio increased overall accuracy by 14.8 percentage points.
  • In Experiment 2 (N = 302), participants were 86% accurate on real PDD videos, 83% on enhanced voice-actor deepfakes, and 72% on enhanced text-to-speech deepfakes; text-to-speech deepfakes were 13 percentage points harder than the Barari et al. 2021 deepfakes they were compared against.
  • In Experiment 4 (N = 206), voice-actor audio increased accuracy by 9.2 percentage points, and video with a voice actor lowered accuracy by 18.7 percentage points relative to audio-only voice-actor conditions.
  • In Experiment 5 (N = 200), relative to transcripts, silent video increased the true positive rate by 10.3 percentage points, audio by 6.3 points, and video by 25 points; priming with the attention-check deepfake increased the false positive rate by 6.3 points.

## Methods

  • The paper reports 5 pre-registered randomized experiments, plus a non-preregistered robustness check (Experiment 1b), using political speech stimuli from the Presidential Deepfakes Dataset and 12 additional videos.
  • Participants were recruited mainly via Prolific for Experiments 1a and 2–5; Experiment 1b used organic web traffic and included 41,313 participants. The main recruited-sample total is 2,215 participants.
  • Main analysis was pre-registered OLS regressions with robust standard errors clustered at the participant level, supplemented by two-sided t tests and Benjamini-Hochberg correction for multiple comparisons.

## Limitations

  • Stated in the paper: the authors say the study focuses on a single context, political speeches, and on a specific combination of manipulation methods (deepfake lip-syncing wav2lip and the DeepFaceLab library), so the results may not generalize to other deepfake types or contexts.
  • Stated in the paper: methodological details for producing the enhanced deepfakes are withheld or only available upon request to reduce misuse risk.
  • Evident from the text: the strongest conclusions are about short political-speech stimuli and judged accuracy in controlled experiments, not about real-world sharing, persuasion, or broader misinformation behavior.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-024-51998-z

Cites

100 of 100 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Accuracy distinguishing real and fabricated speeches across video stimuli. Accuracy across the original Presidential Deepfakes Dataset (PDD) video stimuli in Experiment 1a (N = 2228 observations), the enhanced PDD video stimuli in Experiment 2 (N = 3580 observations), and the non-PDD video stimuli in Experiment 2 (N = 2384 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.
Fig. 1 Accuracy distinguishing real and fabricated speeches across video stimuli. Accuracy across the original Presidential Deepfakes Dataset (PDD) video stimuli in Experiment 1a (N = 2228 observations), the enhanced PDD video stimuli in Experiment 2 (N = 3580 observations), and the non-PDD video stimuli in Experiment 2 (N = 2384 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.CC BY · PubMed Central
Fig. 2 Accuracy distinguishing real and fabricated speeches across media modalities in experiment 1. A Mean accuracy across all permutations of text, audio, and video in Experiment 1a with 501 recruited participants (N = 16,011 observations). B Mean accuracy across all permutations of text, audio, and video in Experiment 1b with 41,313 non-recruited participants (N = 416,901 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.
Fig. 2 Accuracy distinguishing real and fabricated speeches across media modalities in experiment 1. A Mean accuracy across all permutations of text, audio, and video in Experiment 1a with 501 recruited participants (N = 16,011 observations). B Mean accuracy across all permutations of text, audio, and video in Experiment 1b with 41,313 non-recruited participants (N = 416,901 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.CC BY · PubMed Central
Fig. 3 Average accuracy and confidence for all video stimuli in experiment 2. Scatter plot showing participants′ mean accuracy and confidence (N = 5964 observations) across the 60 videos in Experiment 2. PDD indicates videos from the enhanced Presidential Deepfake Dataset, and Other indicates videos are drawn from the same sample used in Barari et al. 2021. VA indicates voice actor deepfakes and TTS indicates text-to-speech deepfakes.
Fig. 3 Average accuracy and confidence for all video stimuli in experiment 2. Scatter plot showing participants′ mean accuracy and confidence (N = 5964 observations) across the 60 videos in Experiment 2. PDD indicates videos from the enhanced Presidential Deepfake Dataset, and Other indicates videos are drawn from the same sample used in Barari et al. 2021. VA indicates voice actor deepfakes and TTS indicates text-to-speech deepfakes.CC BY · PubMed Central
Fig. 4 Accuracy distinguishing real and fabricated speeches across media modalities and base rates in experiment 3. A Mean accuracy across all permutations of text, audio, and video and high and low-base rate conditions in Experiment 3 (N = 19,812 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus. B Low base rate condition in Experiment 3: Mean accuracy across the four modalities (N = 9572 observations). C High base rate condition in Experiment 3: Mean accuracy across the four modalities (N = 10,240 observations).
Fig. 4 Accuracy distinguishing real and fabricated speeches across media modalities and base rates in experiment 3. A Mean accuracy across all permutations of text, audio, and video and high and low-base rate conditions in Experiment 3 (N = 19,812 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus. B Low base rate condition in Experiment 3: Mean accuracy across the four modalities (N = 9572 observations). C High base rate condition in Experiment 3: Mean accuracy across the four modalities (N = 10,240 observations).CC BY · PubMed Central
Fig. 5 Accuracy distinguishing speakers’ and actors’ voices across video stimuli in experiment 4. Mean accuracy across all audio and video in Experiment 4 (N = 3215 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.
Fig. 5 Accuracy distinguishing speakers’ and actors’ voices across video stimuli in experiment 4. Mean accuracy across all audio and video in Experiment 4 (N = 3215 observations). The error bars represent 95% confidence intervals. Dot plots represent the mean accuracy for each video stimulus.CC BY · PubMed Central
Fig. 6 True positive and false positive rate in suspicion of fabrications in experiment 5. A Mean true positive rate (accurate suspicion of a fabrication) across modalities and the attention check deepfake where a man’s face and glasses transform in front of the viewer (N = 1218 observations). B Mean false positive rate (inaccurate suspicion of a fabrication) across modalities (N = 982 observations). The error bars represent 95% confidence intervals. Dot plots represent each video stimulus.
Fig. 6 True positive and false positive rate in suspicion of fabrications in experiment 5. A Mean true positive rate (accurate suspicion of a fabrication) across modalities and the attention check deepfake where a man’s face and glasses transform in front of the viewer (N = 1218 observations). B Mean false positive rate (inaccurate suspicion of a fabrication) across modalities (N = 982 observations). The error bars represent 95% confidence intervals. Dot plots represent each video stimulus.CC BY · PubMed Central

Verification

Claims the paper makes

17 Valid · 0 Not valid · 32 Not checked · 158 Own results and interpretations

102 more claims · 2 with their cited sources checked

Create a free account to see all 207 claims and to verify the 9 references still open.

See all claims — free account
  • Probably not a substantial claim. 76
  • It may state two separate claims as one. 50
  • The quote only partly states this claim. 5

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 101 references

Research works printed but not linked: 5

  1. Peirce, C. S. Peirce on Signs: Writings on Semiotic (UNC Press Books, 1991).
  2. Sankaranarayanan, A., Groh, M., Picard, R. & Lippman, A. The presidential deepfakes dataset. In Proceedings of the AIofAI Workshop at the International Joint Conference on Artificial Intelligence (2021).
  3. Fuller, T.Gnomologia: Adagies and Proverbs; Wise Sentences and Witty Sayings, Ancient and Modern, Foreign and British, vol. 1 (B. Barker, 1732).
  4. Sankaranarayanan, A., Groh, M., Picard, R. & Lippman, A. The presidential deepfakes dataset. In CEUR Workshop Proceedings, vol. 2942, 57–72 (CEUR-WS, 2021).
  5. Groh, M. et al. Stimuli for “Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video”. https://doi.org/10.48550/arXiv.2202.12883 (2024). · DOI

Cited by

See all 89

Metadata

Authors
Groh, M., Sankaranarayanan, A., Singh, N., Kim, D. Y., Lippman, A., & Picard, R.
Affiliations
  • Northwestern University (US)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • MIT Media Lab (US)
Corresponding author
Groh, Matthew
Venue
Nature Communications
Published
2024-09-02
Volume
15
Issue
1
Pages
7629-7629
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4402130146
Cited by
89
References
101
Citations by year
2026: 45 · 2025: 29 · 2024: 12
FWCI
61.40
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
TruePic Research Grant, Kellogg School of Management, MIT Media Lab
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
Keywords
deepfakes, political speeches, misinformation detection, audiovisual cues, media modality, question framing, priming
MeSH terms
Adult, Algorithms, Communication, Female, Humans, Male, Politics, Speech, Video Recording, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Human detection of political speech deepfakes across transcripts, audio, and video. https://app.cochat.ai/papers/10.1038/s41467-024-51998-z
The paper
Groh, M., Sankaranarayanan, A., Singh, N., Kim, D. Y., Lippman, A., & Picard, R. (2024). Human detection of political speech deepfakes across transcripts, audio, and video. Nature Communications. https://doi.org/10.1038/s41467-024-51998-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.