CoChat

Voice-Face Homogeneity Tells Deepfake

Open access
Cheng, Harry, Guo, Yangyang, Wang, Tianyi, Li, Qi, Chang, Xiaojun, Nie, Liqiang · 2024 · ACM Transactions on Multimedia Computing, Communications, and Applications
Cited by 102

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Abstract

Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of these approaches has reached a blockage. In this article, we propose to perform deepfake detection from an unexplored voice-face matching view. Our approach is founded on two supporting points: first, there is a high degree of homogeneity between the voice and face of an individual (i.e., they are highly correlated), and second, deepfake videos often involve mismatched identities between the voice and face due to face-swapping techniques. To this end, we develop a voice-face matching method that measures the matching degree between these two modalities to identify deepfake videos. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets: DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. For instance, our method outperforms the baselines by nearly 2%, achieving an AUC of 86.11% on FakeAVCeleb. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3625231

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 60

  1. 10.1111/j.1467-8659.2004.00799.x · DOI · Paper page
  2. Harry Cheng Yangyang Guo Liqiang Nie Zhiyong Cheng and Mohan S. Kankanhalli. 2023. Sample less learn more: Efficient action recognition via frame feature restoration. arXiv:2307.14866 (2023). · DOI
  3. Harry Cheng, Yangyang Guo, Jianhua Yin, Haonan Chen, Jiafang Wang, and Liqiang Nie. 2021. Audio-driven talking video frame restoration. IEEE Transactions on Multimedia. Early access, October 7, 2021.
  4. Kai Cheng, Xin Liu, Yiu-ming Cheung, Rui Wang, Xing Xu, and Bineng Zhong. 2020. Hearing like seeing: Improving voice-face interactions and associations via adversarial deep semantic matching network. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 448–455.
  5. Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. VoxCeleb2: Deep speaker recognition. In Proceedings of the Conference of the International Speech Communication Association. 1086–1090.
  6. Brian Dolhansky Joanna Bitton Ben Pflaum Jikuo Lu Russ Howes Menglin Wang and Cristian Canton-Ferrer. 2020. The DeepFake Detection Challenge (DFDC) dataset. arXiv:2006.07397 (2020). · arXiv
  7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations.1–12.
  8. 10.1145/3536426 · DOI · Paper page
  9. 10.1145/3422622 · DOI · Paper page
  10. 10.24963/ijcai.2021/98 · DOI
  11. 10.1109/TIP.2021.3128322 · DOI
  12. Zhiqing Guo, Gaobo Yang, Jiyou Chen, and Xingming Sun. 2023. Exposing deepfake face forgeries with guided residuals. IEEE Transactions on Multimedia. Early access, January 18, 2023.
  13. 10.1016/j.cviu.2022.103587 · DOI
  14. 10.1016/j.eswa.2022.119361 · DOI
  15. Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2021. Lips don’t lie: A generalisable and robust approach to face forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5039–5049.
  16. Shota Horiguchi, Naoyuki Kanda, and Kenji Nagamatsu. 2018. Face-voice matching using cross-modal embeddings. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 1011–1019. · DOI
  17. 10.1016/j.cub.2003.09.005 · DOI · Paper page
  18. 10.1109/TPAMI.2020.2970919 · doi:10.1109/tpami.2020.2970919 · Paper page
  19. Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM Transactions on Graphics 35, 4 (2016), Article 94, 8 pages.
  20. Hasam Khalid, Minha Kim, Shahroz Tariq, and Simon S. Woo. 2021. Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors. In Proceedings of the ACM Workshop on Synthetic Multimedia. ACM, New York, NY, 7–15.
  21. Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. 2018. Deep video portraits. ACM Transactions on Graphics 37, 4 (2018), Article 163, 14 pages.
  22. Pavel Korshunov and Sébastien Marcel. 2018. Speaker inconsistency detection in tampered video. In Proceedings of the European Signal Processing Conference. IEEE, Los Alamitos, CA, 2375–2379.
  23. Mohammad Rami Koujan, Michail Christos Doukas, Anastasios Roussos, and Stefanos Zafeiriou. 2020. Head2Head: Video-based neural head synthesis. In Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE, Los Alamitos, CA, 16–23.
  24. John K. Lewis, Imad Eddine Toubal, Helen Chen, Vishal Sandesera, Michael Lomnitz, Zigfried Hampel-Arias, Prasad Calyam, and Kannappan Palaniappan. 2020. Deepfake video detection based on spatial, spectral, and temporal inconsistencies using multimodal deep learning. In Proceedings of the Applied Imagery Pattern Recognition Workshop. IEEE, Los Alamitos, CA, 1–9.
  25. Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. 2020. Advancing high fidelity identity swapping for forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5073–5082.
  26. Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. 2020. Face x-ray for more general face forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5000–5009.
  27. Yidong Li, Wenhua Liu, Yi Jin, and Yuanzhouhan Cao. 2021. SPGAN: Face forgery using spoofing generative adversarial networks. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1–20.
  28. Yuezun Li and Siwei Lyu. 2019. Exposing DeepFake videos by detecting face warping artifacts. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. IEEE, Los Alamitos, CA, 46–52.
  29. Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-DF: A large-scale challenging dataset for DeepFake forensics. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 3204–3213.
  30. Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. 2021. Spatial-phase shallow learning: Rethinking face forgery detection in frequency domain. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 772–781.
  31. Xiaolong Liu, Yang Yu, Xiaolong Li, Yao Zhao, and Guodong Guo. 2023. TCSD: Triple complementary streams detector for comprehensive deepfake detection. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 6 (2023), Article 213, 22 pages.
  32. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations. 1–8.
  33. Bahar Uddin Mahmud, Guan Yue Hong, and Bernard Fong. 2023. A study of human-AI symbiosis for creative work: Recent developments and future directions in deep learning. ACM Transactions on Multimedia Computing, Communications, and Applications 20, 2 (2023), Article 47, 21 pages.
  34. Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. Emotions don’t lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 2823–2832.
  35. Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. 2018. Seeing voices and hearing faces: Cross-modal biometric matching. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 8427–8436.
  36. Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A large-scale speaker identification dataset. In Proceedings of the Conference of the International Speech Communication Association. 2616–2620.
  37. Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2019. Capsule-forensics: Using capsule networks to detect forged images and videos. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. IEEE, Los Alamitos, CA, 2307–2311.
  38. Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. FSGAN: Subject agnostic face swapping and reenactment. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 7183–7192.
  39. Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. FSGAN: Subject agnostic face swapping and reenactment. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 7183–7192.
  40. Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik. 2019. Speech2Face: Learning the face behind a voice. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 7539–7548.
  41. 10.1145/3287309 · DOI
  42. Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. FaceForensics++: Learning to detect manipulated facial images. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 1–11.
  43. Conrad Sanderson and Brian C. Lovell. 2009. Multi-region probabilistic histograms for robust and scalable identity inference. In Proceedings of the International Conference on Biometrics. 199–208.
  44. Huiru Shao, Jing Li, Jia Zhang, Hui Yu, and Jiande Sun. 2020. Eye-based recognition for user identification on mobile devices. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 4 (2020), 1–19. · doi:10.1145/3399659
  45. 10.1007/s00530-021-00837-y · DOI
  46. 10.1007/s11042-021-10989-8 · DOI
  47. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations. 1–10.
  48. Mingxing Tan and Quoc V. Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning. 6105–6114.
  49. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605.
  50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Conference on Neural Information Processing Systems. 5998–6008.
  51. Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2021. HifiFace: 3D shape and semantic prior guided high fidelity face swapping. In Proceedings of the International Joint Conference on Artificial Intelligence. 1136–1142.
  52. Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, and Qingming Huang. 2021. Seeking the shape of sound: An adaptive framework for learning voice-face association. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 16347–16356.
  53. Han Xue, Jun Ling, Anni Tang, Li Song, Rong Xie, and Wenjun Zhang. 2023. High-fidelity face reenactment via identity-matched correspondence learning. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 3 (2023), Article 122, 23 pages.
  54. 10.1145/3472810 · DOI · Paper page
  55. 10.1016/j.knosys.2022.108223 · DOI
  56. Xin Yang, Yuezun Li, and Siwei Lyu. 2019. Exposing deep fakes using inconsistent head poses. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. IEEE, Los Alamitos, CA, 8261–8265.
  57. 10.1109/TIP.2022.3157450 · DOI · Paper page
  58. Yang Yu, Rongrong Ni, Wenjie Li, and Yao Zhao. 2022. Detection of AI-manipulated fake faces via mining generalized features. ACM Transactions on Multimedia Computing, Communications, and Applications 18, 4 (2022), Article 94, 23 pages.
  59. Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. 2021. Multi-attentional deepfake detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 2185–2194.
  60. Yipin Zhou and Ser-Nam Lim. 2021. Joint audio-visual deepfake detection. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 14800–14809.

Cited by

See all 102

Metadata

Authors
Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L.
Venue
ACM Transactions on Multimedia Computing, Communications, and Applications
Published
2023-11-10
Volume
20
Issue
3
Pages
1-22
ISSN
  • 1551-6857 (ISSN-L)
  • 1551-6865
OpenAlex
W4386928847
Cited by
102
References
15
FWCI
9.18
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Voice-Face Homogeneity Tells Deepfake. https://app.cochat.ai/papers/10.1145/3625231
The paper
Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L. (2024). Voice-Face Homogeneity Tells Deepfake. ACM Transactions on Multimedia Computing, Communications, and Applications. https://doi.org/10.1145/3625231

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.