The DOI matches its Crossref record; 100 of the 101 references it cites were found in a registry or an index; 20 of the 75 sources its claims cite were checked for support.
Human detection of political speech deepfakes across transcripts, audio, and video
Open accessAbstract
Recent advances in technology for hyper-realistic visual and audio effects provoke the concern that deepfake videos of political speeches will soon be indistinguishable from authentic video. We conduct 5 pre-registered randomized experiments with N = 2215 participants to evaluate how accurately humans distinguish real political speeches from fabrications across base rates of misinformation, audio sources, question framings with and without priming, and media modalities. We do not find base rates of misinformation have statistically significant effects on discernment. We find deepfakes with audio produced by the state-of-the-art text-to-speech algorithms are harder to discern than the same deepfakes with voice actor audio. Moreover across all experiments and question framings, we find audio and visual information enables more accurate discernment than text alone: human discernment relies more on how something is said, the audio-visual cues, than what is said, the speech content.
Overview
CoChat Summary
## TL;DR
This paper runs 5 pre-registered randomized experiments with 2,215 recruited participants to test how well humans detect fabricated political speeches across text, audio, and video. Overall, people are more accurate when audio and visual cues are available than when judging transcripts alone, and the paper finds no statistically significant effect of misinformation base rate on overall discernment.
## Key findings
- In Experiment 1a (N = 501), accuracy rose with more modalities: 57% for transcripts, 64% for silent video without subtitles, 69% for silent video with subtitles, 81% for audio, 85% for video with audio and subtitles, and 86% for video with audio but no subtitles.
- In Experiment 1b (41,313 participants; not preregistered), the same ordering held, from 43% accuracy for transcripts up to 83% for video with audio but no subtitles.
- In Experiment 3 (N = 1,006), the base rate of fakes did not significantly change overall accuracy, but a high-fake condition increased accuracy on real stimuli by 7.2 percentage points and decreased accuracy on fabricated stimuli by 5.8 percentage points; video with audio increased overall accuracy by 14.8 percentage points.
- In Experiment 2 (N = 302), participants were 86% accurate on real PDD videos, 83% on enhanced voice-actor deepfakes, and 72% on enhanced text-to-speech deepfakes; text-to-speech deepfakes were 13 percentage points harder than the Barari et al. 2021 deepfakes they were compared against.
- In Experiment 4 (N = 206), voice-actor audio increased accuracy by 9.2 percentage points, and video with a voice actor lowered accuracy by 18.7 percentage points relative to audio-only voice-actor conditions.
- In Experiment 5 (N = 200), relative to transcripts, silent video increased the true positive rate by 10.3 percentage points, audio by 6.3 points, and video by 25 points; priming with the attention-check deepfake increased the false positive rate by 6.3 points.
## Methods
- The paper reports 5 pre-registered randomized experiments, plus a non-preregistered robustness check (Experiment 1b), using political speech stimuli from the Presidential Deepfakes Dataset and 12 additional videos.
- Participants were recruited mainly via Prolific for Experiments 1a and 2–5; Experiment 1b used organic web traffic and included 41,313 participants. The main recruited-sample total is 2,215 participants.
- Main analysis was pre-registered OLS regressions with robust standard errors clustered at the participant level, supplemented by two-sided t tests and Benjamini-Hochberg correction for multiple comparisons.
## Limitations
- Stated in the paper: the authors say the study focuses on a single context, political speeches, and on a specific combination of manipulation methods (deepfake lip-syncing wav2lip and the DeepFaceLab library), so the results may not generalize to other deepfake types or contexts.
- Stated in the paper: methodological details for producing the enhanced deepfakes are withheld or only available upon request to reduce misuse risk.
- Evident from the text: the strongest conclusions are about short political-speech stimuli and judged accuracy in controlled experiments, not about real-world sharing, persuasion, or broader misinformation behavior.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
17 Valid · 0 Not valid · 32 Not checked · 158 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Recent advances in technology for algorithmically applying hyper-realistic manipulations to video are simultaneously enabling new forms of interpersonal communication and posing a threat to traditional standards of evidence and trust in media1–8.”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Introduction
What the sources say
- Not checked — no full text of the sourcecites [1] Hancock, J. T. & Bailenson, J. N. The social impact of deepfakes. Cyberpsychol. Behav. Soc. Netw. 24, 149–152 (2021). source
- Not checked — no full text of the sourcecites [2] Chesney, B. & Citron, D. Deep fakes: A looming challenge for privacy, democracy, and national security. Calif. L. Rev. 107, 1753 (2019). source
- Not matchedcites [3] Paris, B. & Donovan, J. Deepfakes and Cheap Fakes. United States of America: Data & Society (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [4] Leibowicz, C., McGregor, S. & Ovadya, A. The Deepfake Detection Dilemma: A Multistakeholder Exploration of Adversarial Dynamics in Synthetic Media (2021). source
- Not matchedcites [5] Agarwal, S. et al. Protecting World Leaders Against Deep Fakes. In CVPR workshops, vol. 1 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [6] Pataranutaporn, P. et al. Ai-generated characters for supporting personalized learning and well-being. Nat. Mach. Intell. 3, 1013–1022 (2021). source
- Not checked — no full text of the sourcecites [7] Guess, A. M. & Lyons, B. A. Misinformation, disinformation, and online propaganda. Social media and democracy: The state of the field, prospects for reform 10–33 (2020). source
- Supportedcites [8] Boháček, M. & Farid, H. Protecting world leaders against deep fakes using facial, gestural, and vocal mannerisms. Proc. Natl. Acad. Sci. USA 119, e2216035119 (2022). source
The Discussion says deep-fake videos are becoming higher quality and easier to create, while posing new threats to trust in what people see and hear online. The Abstract also describes synthetic media as enabling disinformation and making it easier to deny reality. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“people with more analytic cognitive styles worldwide are more accurate at discerning between authentic and fabricated political videos62 and true and false headlines related to COVID-1963.”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Introduction
What the sources say
- Partly supportedcites [62] Appel, M. & Prietzel, F. The detection of political deepfakes. J. Comput. Mediat.Commun. 27, zmac008 (2022). sourceExperimental — accuracy not yet measured
The General discussion reports that higher analytic thinking was associated with successful deepfake detection across two studies. But the Participants and procedure sections describe samples recruited through a German crowdworking service, not worldwide samples, and Study 2 used two fabricated videos rather than a test distinguishing authentic from fabricated videos. The COVID-19 headlines clause carries a different citation and is not attributed to this paper. (LLM, conf=0.88)
- Supportedcites [63] Arechar, A. A. et al. Understanding and reducing online misinformation across 16 countries on six continents. Nat. Hum. Behav. 7, 1502–1513 (2022). source
The 'Who believes misinformation?' section states that in every country participants with a more analytic cognitive style (self-reported analytic thinking and CRT performance) were better able to discern truth from falsehood, with meta-analytic interaction coefficients b=0.037 and b=0.034 (p<0.001). The Abstract confirms the stimulus domain and scope: an experiment in 16 countries across 6 continents (N=34,286) on susceptibility to COVID-19 misinformation, where 'in every country, participants with a more analytic cognitive style... were better at discerning truth from falsehood.' The political-videos half of the citing sentence carries a separate citation (62) and is not part of what this source is offered for. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On a subset of 11 of the 166 videos, Kobis et al. 2021 do not find that people can detect deepfakes better than chance50.”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Introduction
What the sources say
- Partly supportedcites [50] Köbis, N., Doležalová, B. & Soraperra, I. Fooled twice–people cannot detect deepfakes but think they can. Science 24, 103364 (2021). sourceExperimental — accuracy not yet measured
The Results (Detection accuracy; page unavailable) say that guesses were significantly better than chance for only 5 of 16 videos, supporting the claim for the remaining 11 at the individual-video level. The paper reports 16 videos, not the 166 printed in the citing sentence; it also finds overall accuracy above chance. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“people consider a deepfake of a Dutch politician significantly less credible than the real video from which it was adapted36”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Introduction
What the sources say
- Supportedcites [36] Dobber, T., Metoui, N., Trilling, D., Helberger, N. & de Vreese, C. Do (microtargeted) deepfakes have real effects on political attitudes? Int. J. Press Polit. 26, 69–91 (2021). source
The Manipulation Check (p. 11) reports that participants rated the deepfake significantly less credible than the control video (3.70 versus 4.18; p = .01). Table 1 (p. 15) identifies the control as the original video. The authors caution that participants often interpreted credibility broadly, as referring to the politician or politics rather than detecting manipulation. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These results are based on an experiment with a stimuli set that is much larger than most stimuli sets for the psychology of media effects research38 and deepfake detection33,35,36”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Discussion
What the sources say
- Not checked — no full text of the sourcecites [38] Reeves, B., Yeykelis, L. & Cummings, J. J. The use of media in media psychology. Media Psychol. 19, 49–71 (2016). source
- Not matchedcites [33] Barari, S., Lucas, C. & Munger, K. Political Deepfake Videos Misinform the Public, But No More than Other Fake Media. Open Science Framework (2021).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [35] Vaccari, C. & Chadwick, A. Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Soc. Media+ Soc. 6, 2056305120903408 (2020). source
- Partly supportedcites [36] Dobber, T., Metoui, N., Trilling, D., Helberger, N. & de Vreese, C. Do (microtargeted) deepfakes have real effects on political attitudes? Int. J. Press Polit. 26, 69–91 (2021). sourceExperimental — accuracy not yet measured
The Independent Variables section (p. 8) describes a media-effects experiment using one deepfake video and its original, nonmanipulated counterpart, so the paper provides an example of a small stimulus set. It does not establish that most media-effects studies use sets of that size or compare its set with the citing experiment’s set. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
102 more claims · 2 with their cited sources checked
Create a free account to see all 207 claims and to verify the 9 references still open.
- Probably not a substantial claim. 76
- It may state two separate claims as one. 50
- The quote only partly states this claim. 5
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AIgenerated content (e. g., deepfakes) - artificially generated videos, images, and audio designed to mimic real individuals - has become increasingly prevalent on social media, raising critical concerns about their role in spreading misinformation (Hancock and Bailenson 2021; Groh et al. 2024; Vaccari and Chadwick 2020).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Groh et al. 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Human-centered methods, which are dependent on individual judgments, often struggle to accurately identify AI-generated content (Groh et al. 2024; Somoray and Miller 2023; Diel et al. 2024; Bray, Johnson, and Kleinberg 2023).”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Groh et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“The sample of 509 recruited participants is balanced across political identities: 255 recruited participants self-report as Democrats, and the other 246 recruited participants self-report as Republicans.”
§ Human detection of political speech deepfakes across transcripts, audio, and video > Methods > Participants
Reported 509 · recomputed 501
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“- Figure 2”
What Makes Online Content Viral?
Berger, J., & Milkman, K. L. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0353 · Cited by 3,178 · cited 1×
“We have removed the interaction with CRT from the paper to increase clarity and focus on the main results from the additional experiments 2-4.”
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Research works printed but not linked: 5
- Peirce, C. S. Peirce on Signs: Writings on Semiotic (UNC Press Books, 1991).
- Sankaranarayanan, A., Groh, M., Picard, R. & Lippman, A. The presidential deepfakes dataset. In Proceedings of the AIofAI Workshop at the International Joint Conference on Artificial Intelligence (2021).
- Fuller, T.Gnomologia: Adagies and Proverbs; Wise Sentences and Witty Sayings, Ancient and Modern, Foreign and British, vol. 1 (B. Barker, 1732).
- Sankaranarayanan, A., Groh, M., Picard, R. & Lippman, A. The presidential deepfakes dataset. In CEUR Workshop Proceedings, vol. 2942, 57–72 (CEUR-WS, 2021).
- Groh, M. et al. Stimuli for “Human Detection of Political Speech Deepfakes across Transcripts, Audio, and Video”. https://doi.org/10.48550/arXiv.2202.12883 (2024). · DOI
Cited by
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333
Factuality challenges in the era of large language models and opportunities for fact-checking
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00881-z · Cited by 187
“This issue parallels the rapid advancement of deepfake technology in various media formats, including fake videos, manipulated images, altered audio clips and stylized text 96,97 .”
Generative AI and its Transformative Value for Digital Platforms
Wessel, M., Adam, M., Benlian, A., Majchrzak, A., & Thies, F. (2025) · Journal of Management Information Systems · doi:10.1080/07421222.2025.2487315 · Cited by 243
Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown
Simon, F. M., Altay, S., & Mercier, H. (2023) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-127 · Cited by 85
Human performance in detecting deepfakes: A systematic review and meta-analysis of 56 papers
Diel, A., Lalgi, T., Schröter, I. C., MacDorman, K. F., Teufel, M., & Bäuerle, A. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100538 · Cited by 64
Metadata
- Authors
- Groh, M., Sankaranarayanan, A., Singh, N., Kim, D. Y., Lippman, A., & Picard, R.
- Affiliations
- Northwestern University (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- MIT Media Lab (US)
- Corresponding author
- Groh, Matthew
- ORCID
- Venue
- Nature Communications
- Published
- 2024-09-02
- Volume
- 15
- Issue
- 1
- Pages
- 7629-7629
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11368926
- OpenAlex
- W4402130146
- Semantic Scholar
- e940e035579a20d84e52d2b56505d4827d3d4eff
- Cited by
- 89
- References
- 101
- Citations by year
- 2026: 45 · 2025: 29 · 2024: 12
- FWCI
- 61.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- TruePic Research Grant, Kellogg School of Management, MIT Media Lab
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
- Keywords
- deepfakes, political speeches, misinformation detection, audiovisual cues, media modality, question framing, priming
- MeSH terms
- Adult, Algorithms, Communication, Female, Humans, Male, Politics, Speech, Video Recording, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Human detection of political speech deepfakes across transcripts, audio, and video. https://app.cochat.ai/papers/10.1038/s41467-024-51998-z
- The paper
- Groh, M., Sankaranarayanan, A., Singh, N., Kim, D. Y., Lippman, A., & Picard, R. (2024). Human detection of political speech deepfakes across transcripts, audio, and video. Nature Communications. https://doi.org/10.1038/s41467-024-51998-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.