CoChat
Sign Up
PreprintCited by 116Digital Media Forensic DetectionMisinformation and Its ImpactsBiometric Identification and Security

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

Open access
Khalid, Hasam, Tariq, Shahroz, Kim, Minha, Woo, Simon S.·2021·arXiv
DOI 10.48550/arxiv.2108.05080arXiv 2108.05080
Open this paper in CoChat — free accountOpen PDF

Abstract

While the significant advancements have made in the generation of deepfakes using deep learning technologies, its misuse is a well-known issue now. Deepfakes can cause severe security and privacy issues as they can be used to impersonate a person's identity in a video by replacing his/her face with another person's face. Recently, a new problem of generating synthesized human voice of a person is emerging, where AI-based deep learning models can synthesize any person's voice requiring just a few seconds of audio. With the emerging threat of impersonation attacks using deepfake audios and videos, a new generation of deepfake detectors is needed to focus on both video and audio collectively. To develop a competent deepfake detector, a large amount of high-quality data is typically required to capture real-world (or practical) scenarios. Existing deepfake datasets either contain deepfake videos or audios, which are racially biased as well. As a result, it is critical to develop a high-quality video and audio deepfake dataset that can be used to detect both audio and video deepfakes simultaneously. To fill this gap, we propose a novel Audio-Video Deepfake dataset, FakeAVCeleb, which contains not only deepfake videos but also respective synthesized lip-synced fake audios. We generate this dataset using the most popular deepfake generation methods. We selected real YouTube videos of celebrities with four ethnic backgrounds to develop a more realistic multimodal dataset that addresses racial bias, and further help develop multimodal deepfake detectors. We performed several experiments using state-of-the-art detection methods to evaluate our deepfake dataset and demonstrate the challenges and usefulness of our multimodal Audio-Video deepfake dataset.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2108.05080

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • FakeAVCeleb dataset uses only one method (SV2TTS).

    “For FakeAVCeleb dataset, there is only one method used, thus the EER results are very similar for all folds.”

    From Attack Agnostic Dataset: Towards Generalization and Stabilization of Audio DeepFake Detection (2022) · cites this paper as [14]

    Section 3.1 (p. 7) explicitly says FakeAVCeleb uses four deepfake-generation methods: FaceSwap, FSGAN, Wav2Lip, and SV2TTS. SV2TTS is the voice-cloning method, but it is not the dataset’s only method. The cited paper does not establish the proposed explanation for similar EER results across folds. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Few multimodal datasets containing both counterfeited video and audio tracks have been proposed: DFDC, FakeAVCeleb, and DeepfakeTIMIT with TIMIT-TTS.

    “In the recent years, few multimodal datasets have been proposed, containing both counterfeited video and audio tracks.”

    From A Robust Approach to Multimodal Deepfake Detection (2023) · cites this paper as [43]

    The Introduction (p. 2) identifies DFDC as a dataset containing deepfake video and synthesized audio, and describes FakeAVCeleb as containing both. That supports the citing sentence’s general point that such datasets were few. But Background and Motivation (p. 3) says Deepfake TIMIT’s audio was real, not fake; our copy does not identify a TIMIT-TTS variant, so the claim’s three-dataset list is not fully supported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The FakeAVCeleb dataset contains multimodal manipulated videos belonging to five different manipulation categories with diverse ethnicity and gender.

    “The overall dataset contains multimodal manipulated videos belonging to five different manipulation categories with diverse ethnicity and gender.”

    From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [36]

    The Introduction (p. 2) describes FakeAVCeleb as a multimodal audio-video deepfake dataset, and Dataset Collection and Generation (p. 4) describes its ethnic and gender diversity. But the paper identifies three types of audio-video deepfakes (Introduction, p. 2) and four audio-video combinations (Dataset Collection and Generation, p. 4), not five manipulation categories. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

  • The FakeAVCeleb dataset includes multimodal deepfake subsets generated using various audio-visual manipulation techniques such as identity swap (Faceswap, Faceswap-wav2lip), expression swap (Fsgan, Fsgan-wav2lip), lip-syncing (wav2lip), and cloned speech (RTVC).

    “The FakeAVCeleb dataset includes a number of multimodal deepfake subsets generated using various audio-visual manipulation techniques such as identity swap (Faceswap, Faceswap-wav2lip), expression swap (Fsgan, Fsgan-wav2lip), lip-syncing (wav2lip) and cloned speech (RTVC).”

    From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [36]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Various multi-modal fusion techniques have been proposed to enhance robustness by integrating visual, audio, and textual information.

    “Various multi-modal fusion techniques [10], [11], [12], [13] have been proposed to enhance robustness by integrating visual, audio, and textual information.”

    From Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges (2024) · cites this paper as [11]

    The Conclusion (p. 10) and C.4 Multimodal Results (p. 21) describe audio-video deepfake data and evaluations of multimodal methods. But the paper does not describe a method integrating visual, audio, and textual information together: B.1 (p. 18) says the text-processing block of an existing method was replaced for its audio-video experiment. C.4 reports limited performance rather than showing enhanced robustness. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 58 · Cited by 116

Research works printed but not linked: 46

  1. [1][1] MSridevi, C Mala, and Siddhant Sanyam. Comparative study of image forgery and copy-move techniques. Advances in Computer Science, Engineering & Applications , pages 715-723, 2012. -
  2. [2][2] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 , 2014. - · arXiv
  3. [4][4] Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech synthesis, 2019. arXiv:1806.04558 . - · arXiv
  4. [6][6] Sercan O. Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples, 2018. arXiv:1802.06006 . - · arXiv
  5. [7][7] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In European Conference on Computer Vision , pages 716-731. Springer, 2020. -
  6. [12][12] Shahroz Tariq, Sowon Jeon, and Simon S Woo. Am I a Real or Fake Celebrity? Measuring Commercial Face Recognition Web APIs under Deepfake Impersonation Attack. arXiv preprint arXiv:2103.00847 , 2021. - · arXiv
  7. [13][13] Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. Use of a capsule network to detect fake images and videos. arXiv preprint arXiv:1910.12467 , 2019. - · arXiv
  8. [14][14] Xin Yang, Yuezun Li, and Siwei Lyu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8261-8265. IEEE, 2019. -
  9. [15][15] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS) , pages 1-7. IEEE, 2018. -
  10. [17][17] Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. Protecting world leaders against deep fakes. In CVPR Workshops , pages 38-45, 2019. -
  11. [18][18] David Güera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) , pages 1-6. IEEE, 2018. -
  12. [19][19] Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In ictu oculi: Exposing ai generated fake face videos by detecting eye blinking. arXiv preprint arXiv:1806.02877 , 2018. - · arXiv
  13. [20][20] Shahroz Tariq, Sangyup Lee, Hoyoung Kim, Youjin Shin, and Simon S. Woo. Detecting Both Machine and Human Created Fake Face Images In the Wild. In Proceedings of the 2nd International Workshop on Multimedia Privacy and Security , MPS '18, page 81-87, New York, NY, USA, 2018. Association for Computing Machinery. doi:10.1145/3267357.3267367 . - · DOI
  14. [21][21] Shahroz Tariq, Sangyup Lee, Hoyoung Kim, Youjin Shin, and Simon S. Woo. GAN is a Friend or Foe? A Framework to Detect Various Fake Face Images. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing , SAC '19, page 1296-1303, New York, NY, USA, 2019. Association for Computing Machinery. doi:10.1145/3297280.3297410 . - · DOI
  15. [23][23] Sangyup Lee, Shahroz Tariq, Youjin Shin, and Simon S. Woo. Detecting handcrafted facial image manipulations and GAN-generated facial images using Shallow-FakeFaceNet. Applied Soft Computing , 105:107256, 2021. URL: https://www.sciencedirect. com/science/article/pii/S1568494621001794 , doi:https://doi.org/10.1016/j. asoc.2021.107256 . - · DOI
  16. [24][24] Sangyup Lee, Shahroz Tariq, Junyaup Kim, and Simon S. Woo. TAR: Generalized Forensic Framework to Detect Deepfakes Using Weakly Supervised Learning. In Audun Jøsang, Lynn Futcher, and Janne Hagen, editors, ICT Systems Security and Privacy Protection , pages 351-366, Cham, 2021. Springer International Publishing. -
  17. [25][25] Hyeonseong Jeon, Youngoh Bang, and Simon S Woo. FakeTalkerDetect: Effective and Practical Realistic Neural Talking Head Detection with a Highly Unbalanced Dataset. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 0-0, 2019. -
  18. [27][27] Hyeonseong Jeon, Young Oh Bang, Junyaup Kim, and Simon Woo. T-GD: Transferable GANgenerated Images Detection Framework. In International Conference on Machine Learning , pages 4746-4761. PMLR, 2020. -
  19. [28][28] Shahroz Tariq, Sangyup Lee, and Simon S Woo. A Convolutional LSTM based Residual Network for Deepfake Video Detection. arXiv preprint arXiv:2009.07480 , 2020. - · arXiv
  20. [29][29] Minha Kim, Shahroz Tariq, and Simon S. Woo. FReTAL: Generalizing Deepfake Detection Using Knowledge Distillation and Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 1001-1012, June 2021. -
  21. [30][30] Minha Kim, Shahroz Tariq, and Simon S. Woo. CoReD: Generalizing Fake Media Detection with Continual Representation Using Distillation. In Proceedings of the 29th ACM International Conference on Multimedia , MM '21, page 337-346, New York, NY, USA, 2021. Association for Computing Machinery. doi:10.1145/3474085.3475535 . - · DOI
  22. [31][31] Junyaup Kim, Siho Han, and Simon S Woo. Classifying Genuine Face images from Disguised Face Images. In 2019 IEEE International Conference on Big Data (Big Data) , pages 6248-6250. IEEE, 2019. -
  23. [36][36] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2019. -
  24. [37][37] Patrick Kwon, Jaeseong You, Gyuhyeon Nam, Sungwoo Park, and Gyeongsu Chae. Kodf: A large-scale korean deepfake detection dataset. arXiv preprint arXiv:2103.10094 , 2021. - · arXiv
  25. [38][38] Vineet Mehta, Parul Gupta, Ramanathan Subramanian, and Abhinav Dhall. Fakebuster: A deepfakes detection tool for video conferencing scenarios. In 26th International Conference on Intelligent User Interfaces , pages 61-63, 2021. -
  26. [39][39] Binh M Le and Simon S Woo. Add: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images. arXiv preprint arXiv:2112.03553 , 2021. - · arXiv
  27. [42][42] Iain McCowan, Jean Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, Vasilis Karaiskos, et al. The ami meeting corpus. In Proceedings of the 5th international conference on methods and techniques in behavioral research , volume 88, page 100. Citeseer, 2005. -
  28. [43][43] Binh M Le and Simon S Woo. Exploring the asynchronous of the frequency spectra of gangenerated facial images. arXiv preprint arXiv:2112.08050 , 2021. - · arXiv
  29. [44][44] Shahroz Tariq, Sowon Jeon, and Simon S Woo. Am i a real or fake celebrity? measuring commercial face recognition web apis under deepfake impersonation attack. arXiv preprint arXiv:2103.00847 , 2021. - · arXiv
  30. [49][49] Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Xiaobai Li, Feng Zhou, and Guoying Zhao. Searching central difference convolutional networks for face anti-spoofing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5295-5305, 2020. -
  31. [50][50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  32. [51][51] Falko Matern, Christian Riess, and Marc Stamminger. Exploiting visual artifacts to expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) , pages 83-92. IEEE, 2019. -
  33. [52][52] Pavel Korshunov and Sébastien Marcel. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685 , 2018. - · arXiv
  34. [53][53] Jiameng Pu, Neal Mangaokar, Lauren Kelly, Parantapa Bhattacharya, Kavya Sundaram, Mobin Javed, Bolun Wang, and Bimal Viswanath. Deepfake videos in the wild: Analysis and detection. arXiv preprint arXiv:2103.04263 , 2021. - · arXiv
  35. [54][54] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Faceshifter: Towards high fidelity and occlusion aware face swapping. arXiv preprint arXiv:1912.13457 , 2019. - · arXiv
  36. [57][57] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUI) , 3(1), 2019. -
  37. [59][59] Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , Oct 2017. -
  38. [60][60] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 , 2017. - · arXiv
  39. [61][61] Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7184-7193, 2019. -
  40. [62][62] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia , pages 484-492, 2020. -
  41. [65][65] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5001-5010, 2020. -
  42. [66][66] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European Conference on Computer Vision , pages 86-103. Springer, 2020. -
  43. [67][67] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: A generalisable and robust approach to face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5039-5049, 2021. -
  44. [68][68] Hasam Khalid, Minha Kim, Shahroz Tariq, and Simon S. Woo. Evaluation of an Audio-Video Multimodal Deepfake Dataset Using Unimodal and Multimodal Detectors. In Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection , ADGD '21, page 7-15, New York, NY, USA, 2021. Association for Computing Machinery. doi:10.1145/3476099.3484315 . - · DOI · Paper page
  45. [69][69] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675 , 2016. - · arXiv
  46. [71][71] Béatrice Daille, Éric Gaussier, and Jean-Marc Langé. Towards automatic extraction of monolingual and bilingual terminology. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics , 1994. -

Authors
Khalid, H., Tariq, S., Kim, M., & Woo, S. S.
Affiliations
Sungkyunkwan University (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2108.05080
arXiv
2108.05080
OpenAlex
W3190486916
Cited by
116
References
58
Citations by year
2026: 2 · 2025: 21 · 2024: 43 · 2023: 38 · 2022: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • National Research Foundation
  • Sungkyunkwan University (2019-0-00421)
  • National Research Foundation of Korea (2020R1C1C1006004, 2019-0-00421)
  • Ministry of Science and ICT, South Korea (2019-0-00421, 2019-0-01343)
  • Institute for Information and Communications Technology Promotion (2019-0-01343, 2019‐0‐00421, 2021-0-00017)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
deepfake detection, synthetic speech, face swapping, deepfake audio

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review