CoChat
Sign Up
PreprintCited by 9Digital Media Forensic DetectionSpeech Recognition and SynthesisMisinformation and Its Impacts

The DOI matches its DataCite record.

The DeepSpeak Dataset

Open access
Barrington, Sarah, Bohacek, Maty, Farid, Hany·2024·arXiv
DOI 10.48550/arxiv.2408.05366arXiv 2408.05366
Open this paper in CoChat — free accountOpen PDF

Abstract

Deepfakes represent a growing concern across domains such as disinformation, fraud, and non-consensual media. In particular, the rise of video conference and identity-driven attacks in high-stakes scenarios--such as impostor hiring--demands new forensic resources. Despite significant efforts to develop robust detection classifiers to distinguish the real from the fake, commonly used training datasets remain inadequate: relying on low-quality and outdated deepfake generators, consisting of content scraped from online repositories without participant consent, lacking in multimodal coverage, and rarely employing identity-matching protocols to ensure realistic fakes. To overcome these limitations, we present the DeepSpeak dataset, a diverse and multimodal dataset comprising over 100 hours of authentic and deepfake audiovisual content, specifically focused on the challenging and diverse talking heads context. We contribute: i) more than 50 hours of real, self-recorded data collected from 500 diverse and consenting participants, ii) more than 50 hours of state-of-the-art audio and visual deepfakes generated using 14 video synthesis engines and three voice cloning engines, and iii) an embedding-based, identity-matching approach to ensure the creation of convincing, high-quality identity face swaps that realistically simulate adversarial deepfake attacks. We also perform large-scale evaluations of state-of-the-art deepfake detectors and show that, without retraining, these detectors fail to generalize to this DeepSpeak dataset, highlighting the importance of a large and diverse dataset containing deepfakes from the latest generative-AI tools.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.05366

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DeepSpeak 1.0 is a large-scale dataset featuring real and deepfake footage of individuals speaking and gesturing in front of webcams, including 17 h of real video from 220 diverse individuals and over 26 h of deepfake footage.

    “DeepSpeak 1.0 [58] is a large-scale dataset featuring real and deepfake footage of individuals speaking and gesturing in front of webcams. The dataset includes 17 h of real video footage from 220 diverse individuals and over 26 h of deepfake footage generated using state-of-the-art face-swap and lip-sync techniques.”

    From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [58]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 50

  1. Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):8717-8727, 2018. 17
  2. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakho- tia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. XLS-R: Self-supervised cross- lingual speech representation learning at scale. In Interspeech, pages 2278-2282, 2022. 6
  3. Sarah Barrington, Romit Barua, Gautham Koorma, and Hany Farid. Single and multi-speaker cloned voice detection: From perceptual to learned features. In IEEE International Work- shop on Information Forensics and Security, pages 1-6. IEEE, 2023. 6
  4. Sarah Barrington, Emily A Cooper, and Hany Farid. Peo- ple are poorly equipped to detect AI-powered voice clones. Scientific Reports, 15(1):11004, 2025. 1
  5. Matyas Bohacek and Hany Farid. Lost in translation: Lip- sync deepfake detection from audio-video mismatch. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4315-4323, 2024. 7
  6. Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. Simswap: An efficient framework for high fidelity face swapping. In Proceedings of the 28th ACM international conference on multimedia, pages 2003-2011, 2020. 16
  7. Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. VideoRetalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia, pages 1-9, 2022. 17
  8. Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep speaker recognition. arXiv:1806.05622, 2018. 17 · arXiv
  9. Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recog- nition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019. 4
  10. Michelle L Ding and Harini Suresh. The malicious techni- cal ecosystem: Exposing limitations in technical governance of ai-generated non-consensual intimate images of adults. arXiv:2504.17663, 2025. 1 · arXiv
  11. Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deep- fake detection challenge (DFDC) dataset. arXiv:2006.07397, 2020. 14 · arXiv
  12. Emilio Ferrara. Charting the landscape of nefarious uses of generative artificial intelligence for online election interfer- ence. arXiv:2406.01862, 2024. 1 · arXiv
  13. J. H. Frank and L. Schönherr. WaveFake: A Data Set to Facilitate Audio DeepFake Detection. In Proceedings of the 35th Conference on Neural Information Processing Sys- tems (NeurIPS), Datasets and Benchmarks Track, pages 1-18, 2021. 14
  14. Matthew Groh, Ziv Epstein, Chaz Firestone, and Rosalind Picard. Deepfake detection by human crowds, machines, and machine-informed crowds. Proceedings of the National Academy of Sciences, 119(1):e2110013119, 2022. 1
  15. Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ig- nacio Lopez Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech syn- thesis. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 4485-4495, Red Hook, NY, USA, 2018. Curran Associates Inc. 14
  16. Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans. AASIST: Audio anti-spoofing using integrated spectro- temporal graph attention networks. In IEEE International Conference on Acoustics, Ppeech and Signal Processing, pages 6367-6371, 2022. 6, 30
  17. Hasam Khalid, Shahroz Tariq, Minha Kim, and Simon S Woo. FakeAVCeleb: A novel audio-video multimodal deepfake dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 14
  18. Nithin Rao Koluguri, Taejin Park, and Boris Ginsburg. Ti- taNet: Neural model for speaker representation with 1D depth- wise separable convolutions and global context. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8102-8106. IEEE, 2022. 6
  19. Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar. Towards automatic face-to-face translation. In 27th ACM International Confer- ence on Multimedia, pages 1428-1436, 2019. 17
  20. Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-DF: A large-scale challenging dataset for deepfake forensics. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3207-3216, 2020. 6, 14
  21. Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. BlendGAN: Implicitly GAN blending for arbitrary stylized face generation. Advances in Neural Information Processing Systems, 34:29710-29722, 2021. 17
  22. Weifeng Liu, Tianyi She, Jiawei Liu, Boheng Li, Dongyu Yao, and Run Wang. Lips are lying: Spotting the temporal inconsistency between audio and visual in lip-syncing deep- fakes. Advances in Neural Information Processing Systems, 37:91131-91155, 2024. 5, 6
  23. Xuechen Liu, Xin Wang, Md Sahidullah, Jose Patino, Héctor Delgado, Tomi Kinnunen, Massimiliano Todisco, Junichi Yamagishi, Nicholas Evans, Andreas Nautsch, and Kong Aik Lee. Asvspoof 2021: Towards spoofed and deepfake speech detection in the wild. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 31:2507-2522, 2023. 14
  24. Steven R Livingstone and Frank A Russo. The Ryerson audio- visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018. 17
  25. Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Me- diaPipe: A framework for building perception pipelines. arXiv:1906.08172, 2019. 5, 18 · arXiv
  26. Soumik Mukhopadhyay, Saksham Suri, Ravi Teja Gadde, and Abhinav Shrivastava. Diff2Lip: Audio conditioned diffusion models for lip-synchronization. In IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pages 5292-5302, 2024. 17
  27. Arsha Nagrani, Joon Son Chung, and Andrew Zisser- man. VoxCeleb: A large-scale speaker identification dataset. arXiv:1706.08612, 2017. 17 · arXiv
  28. Sophie J. Nightingale and Hany Farid. AI-synthesized faces are indistinguishable from real faces and more trustworthy. Proceedings of the National Academy of Sciences, 119(8): e2120481119, 2022. 1
  29. Alessandro Pianese, Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva. Training-free deepfake voice recognition by leveraging large-scale pre-trained models. In ACM Work- shop on Information Hiding and Multimedia Security, page 289-294, New York, NY, USA, 2024. 6
  30. KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In 28th ACM International Conference on Multimedia, pages 484-492, 2020. 17
  31. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 3
  32. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. In International Con- ference on Machine Learning, pages 28492-28518. PMLR, 2023. 17
  33. Xingyu Ren, Alexandros Lattas, Baris Gecer, Jiankang Deng, Chao Ma, and Xiaokang Yang. Facial geometric detail re- covery via implicit representation. In IEEE 17th Interna- tional Conference on Automatic Face and Gesture Recogni- tion, 2023. 15, 16
  34. Ernst H Rothauser. Ieee recommended practice for speech quality measurements. IEEE Transactions on Audio and Electroacoustics, 17(3):225-246, 1969. 4
  35. Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi Saruwatari. JSUT corpus: Free large-scale Japanese speech corpus for end-to-end speech synthesis, 2017. arXiv preprint. 14
  36. Kim Sung-Bin, Lee Chae-Yeon, Gihun Son, Oh Hyun-Bin, Janghoon Ju, Suekyeong Nam, and Tae-Hyun Oh. MultiTalk: Enhancing 3D talking head generation across languages with multilingual video dataset. arXiv:2406.14272, 2024. 17 · arXiv
  37. Hemlata Tak, Jee-Weon Jung, Jose Patino, Madhu Kam- ble, Massimiliano Todisco, and Nicholas Evans. End-to- end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection. arXiv:2107.12710, 2021. 6 · arXiv
  38. Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-end anti-spoofing with RawNet2. In IEEE International Con- ference on Acoustics, Speech and Signal Processing, pages 6369-6373, 2021. 6
  39. Cristian Vaccari and Andrew Chadwick. Deepfakes and disin- formation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social media+ society, 6(1):2056305120903408, 2020. 1
  40. Marco Viola and Cristina Voto. Designed to abuse? deep- fakes and the non-consensual diffusion of intimate images. Synthese, 201(1):30, 2023. 1
  41. Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In European Conference on Com- puter Vision, pages 700-717. Springer, 2020. 17
  42. Zhouxia Wang, Jiawei Zhang, Tianshui Chen, Wenping Wang, and Ping Luo. RestoreFormer++: Towards real-world blind face restoration from undegraded key-value pairs. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023. 16
  43. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1-5. IEEE, 2023. 6
  44. Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. VFHQ: A high-quality dataset and benchmark for video face super-resolution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657-666, 2022. 17
  45. Zhiyuan Yan, Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu, Donghao Luo, Chengjie Wang, Shouhong Ding, Yunsheng Wu, et al. DF40: Toward next-generation deepfake detection. arXiv:2406.13495, 2024. 14 · arXiv
  46. Shuang Yang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang, Jingyun Xiao, Keyu Long, Shiguang Shan, and Xilin Chen. LRW-1000: A naturally-distributed large- scale benchmark for lip reading in the wild. In IEEE Interna- tional Conference on Automatic Face & Gesture Recognition, pages 1-8. IEEE, 2019. 17
  47. Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661-3670, 2021. 17
  48. Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. Advances in Neural Information Process- ing Systems, 35:30599-30611, 2022. 16
  49. Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022. 16, 17
  50. Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In European Conference on Computer Vision, pages 650-667. Springer, 2022.

Cited by

Sorted by citations

Metadata

Authors
Barrington, S., Bohacek, M., & Farid, H.
Affiliations
University of California, Berkeley (US)
ORCID
Farid, Hany
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.05366
arXiv
2408.05366
OpenAlex
W4402952646
Cited by
9
References
0
Citations by year
2026: 3 · 2025: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
deepfake detection, talking heads, face swapping, voice cloning, identity matching, generative AI tools, disinformation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review