Voice-Face Homogeneity Tells Deepfake
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of these approaches has reached a blockage. In this article, we propose to perform deepfake detection from an unexplored voice-face matching view. Our approach is founded on two supporting points: first, there is a high degree of homogeneity between the voice and face of an individual (i.e., they are highly correlated), and second, deepfake videos often involve mismatched identities between the voice and face due to face-swapping techniques. To this end, we develop a voice-face matching method that measures the matching degree between these two modalities to identify deepfake videos. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets: DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. For instance, our method outperforms the baselines by nearly 2%, achieving an AUC of 86.11% on FakeAVCeleb. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_6 · Cited by 1,058
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
Matern, F., Riess, C., & Stamminger, M. (2019) · 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2019.00020 · Cited by 824
FSGAN: Subject Agnostic Face Swapping and Reenactment
Nirkin, Y., Keller, Y., & Hassner, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00728 · Cited by 706
DeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 11×
“We conduct extensive experiments over three widely exploited deepfake datasets - DFDC, FakeAVCeleb [24], and DeepfakeTIMIT [25].”
MethodInfluentialTwo-Branch Recurrent Network for Isolating Deepfakes in Videos
Masi, I., Killekar, A., Mascarenhas, R. M., Gurudatt, S. P., & AbdAlmageed, W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_39 · Cited by 471
Face swapping: automatically replacing faces in photographs
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., & Nayar, S. K. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360638 · Cited by 333
Research works printed but not linked: 58
- 10.1111/j.1467-8659.2004.00799.x · DOI · Paper page
- Harry Cheng Yangyang Guo Liqiang Nie Zhiyong Cheng and Mohan S. Kankanhalli. 2023. Sample less learn more: Efficient action recognition via frame feature restoration. arXiv:2307.14866 (2023). · DOI
- Harry Cheng, Yangyang Guo, Jianhua Yin, Haonan Chen, Jiafang Wang, and Liqiang Nie. 2021. Audio-driven talking video frame restoration. IEEE Transactions on Multimedia. Early access, October 7, 2021.
- Kai Cheng, Xin Liu, Yiu-ming Cheung, Rui Wang, Xing Xu, and Bineng Zhong. 2020. Hearing like seeing: Improving voice-face interactions and associations via adversarial deep semantic matching network. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 448–455.
- Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. VoxCeleb2: Deep speaker recognition. In Proceedings of the Conference of the International Speech Communication Association. 1086–1090.
- Brian Dolhansky Joanna Bitton Ben Pflaum Jikuo Lu Russ Howes Menglin Wang and Cristian Canton-Ferrer. 2020. The DeepFake Detection Challenge (DFDC) dataset. arXiv:2006.07397 (2020). · arXiv
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations.1–12.
- 10.1145/3536426 · DOI · Paper page
- 10.1145/3422622 · DOI · Paper page
- 10.24963/ijcai.2021/98 · DOI
- 10.1109/TIP.2021.3128322 · DOI
- Zhiqing Guo, Gaobo Yang, Jiyou Chen, and Xingming Sun. 2023. Exposing deepfake face forgeries with guided residuals. IEEE Transactions on Multimedia. Early access, January 18, 2023.
- 10.1016/j.cviu.2022.103587 · DOI
- 10.1016/j.eswa.2022.119361 · DOI
- Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2021. Lips don’t lie: A generalisable and robust approach to face forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5039–5049.
- Shota Horiguchi, Naoyuki Kanda, and Kenji Nagamatsu. 2018. Face-voice matching using cross-modal embeddings. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 1011–1019. · DOI
- 10.1016/j.cub.2003.09.005 · DOI · Paper page
- 10.1109/TPAMI.2020.2970919 · doi:10.1109/tpami.2020.2970919 · Paper page
- Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM Transactions on Graphics 35, 4 (2016), Article 94, 8 pages.
- Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. 2018. Deep video portraits. ACM Transactions on Graphics 37, 4 (2018), Article 163, 14 pages.
- Pavel Korshunov and Sébastien Marcel. 2018. Speaker inconsistency detection in tampered video. In Proceedings of the European Signal Processing Conference. IEEE, Los Alamitos, CA, 2375–2379.
- Mohammad Rami Koujan, Michail Christos Doukas, Anastasios Roussos, and Stefanos Zafeiriou. 2020. Head2Head: Video-based neural head synthesis. In Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE, Los Alamitos, CA, 16–23.
- John K. Lewis, Imad Eddine Toubal, Helen Chen, Vishal Sandesera, Michael Lomnitz, Zigfried Hampel-Arias, Prasad Calyam, and Kannappan Palaniappan. 2020. Deepfake video detection based on spatial, spectral, and temporal inconsistencies using multimodal deep learning. In Proceedings of the Applied Imagery Pattern Recognition Workshop. IEEE, Los Alamitos, CA, 1–9.
- Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. 2020. Advancing high fidelity identity swapping for forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5073–5082.
- Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. 2020. Face x-ray for more general face forgery detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 5000–5009.
- Yidong Li, Wenhua Liu, Yi Jin, and Yuanzhouhan Cao. 2021. SPGAN: Face forgery using spoofing generative adversarial networks. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1–20.
- Yuezun Li and Siwei Lyu. 2019. Exposing DeepFake videos by detecting face warping artifacts. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. IEEE, Los Alamitos, CA, 46–52.
- Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-DF: A large-scale challenging dataset for DeepFake forensics. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 3204–3213.
- Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. 2021. Spatial-phase shallow learning: Rethinking face forgery detection in frequency domain. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 772–781.
- Xiaolong Liu, Yang Yu, Xiaolong Li, Yao Zhao, and Guodong Guo. 2023. TCSD: Triple complementary streams detector for comprehensive deepfake detection. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 6 (2023), Article 213, 22 pages.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations. 1–8.
- Bahar Uddin Mahmud, Guan Yue Hong, and Bernard Fong. 2023. A study of human-AI symbiosis for creative work: Recent developments and future directions in deep learning. ACM Transactions on Multimedia Computing, Communications, and Applications 20, 2 (2023), Article 47, 21 pages.
- Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. Emotions don’t lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the ACM International Conference on Multimedia. ACM, New York, NY, 2823–2832.
- Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. 2018. Seeing voices and hearing faces: Cross-modal biometric matching. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 8427–8436.
- Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A large-scale speaker identification dataset. In Proceedings of the Conference of the International Speech Communication Association. 2616–2620.
- Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2019. Capsule-forensics: Using capsule networks to detect forged images and videos. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. IEEE, Los Alamitos, CA, 2307–2311.
- Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. FSGAN: Subject agnostic face swapping and reenactment. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 7183–7192.
- Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik. 2019. Speech2Face: Learning the face behind a voice. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 7539–7548.
- 10.1145/3287309 · DOI
- Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. FaceForensics++: Learning to detect manipulated facial images. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 1–11.
- Conrad Sanderson and Brian C. Lovell. 2009. Multi-region probabilistic histograms for robust and scalable identity inference. In Proceedings of the International Conference on Biometrics. 199–208.
- Huiru Shao, Jing Li, Jia Zhang, Hui Yu, and Jiande Sun. 2020. Eye-based recognition for user identification on mobile devices. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 4 (2020), 1–19. · doi:10.1145/3399659
- 10.1007/s00530-021-00837-y · DOI
- 10.1007/s11042-021-10989-8 · DOI
- Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations. 1–10.
- Mingxing Tan and Quoc V. Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning. 6105–6114.
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Conference on Neural Information Processing Systems. 5998–6008.
- Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2021. HifiFace: 3D shape and semantic prior guided high fidelity face swapping. In Proceedings of the International Joint Conference on Artificial Intelligence. 1136–1142.
- Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, and Qingming Huang. 2021. Seeking the shape of sound: An adaptive framework for learning voice-face association. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 16347–16356.
- Han Xue, Jun Ling, Anni Tang, Li Song, Rong Xie, and Wenjun Zhang. 2023. High-fidelity face reenactment via identity-matched correspondence learning. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 3 (2023), Article 122, 23 pages.
- 10.1145/3472810 · DOI · Paper page
- 10.1016/j.knosys.2022.108223 · DOI
- Xin Yang, Yuezun Li, and Siwei Lyu. 2019. Exposing deep fakes using inconsistent head poses. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. IEEE, Los Alamitos, CA, 8261–8265.
- 10.1109/TIP.2022.3157450 · DOI · Paper page
- Yang Yu, Rongrong Ni, Wenjie Li, and Yao Zhao. 2022. Detection of AI-manipulated fake faces via mining generalized features. ACM Transactions on Multimedia Computing, Communications, and Applications 18, 4 (2022), Article 94, 23 pages.
- Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. 2021. Multi-attentional deepfake detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 2185–2194.
- Yipin Zhou and Ser-Nam Lim. 2021. Joint audio-visual deepfake detection. In Proceedings of the International Conference on Computer Vision. IEEE, Los Alamitos, CA, 14800–14809.
Cited by
AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake
Yang, W., Zhou, X., Chen, Z., Guo, B., Ba, Z., Xia, Z., Cao, X., & Ren, K. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3262148 · Cited by 198
Ilyas, H., Javed, A., & Malik, K. M. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110124 · Cited by 186
Deepfake Detection: A Comprehensive Survey from the Reliability Perspective
Wang, T., Liao, X., Chow, K. P., Lin, X., & Wang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3699710 · Cited by 107
Self-citationNoise Based Deepfake Detection via Multi-Head Relative-Interaction
Wang, T., & Chow, K. P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26701 · Cited by 93
Self-citationMultimodaltrace: Deepfake Detection using Audiovisual Representation Learning
Anas Raza, M., & Mahmood Malik, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00106 · Cited by 63
PVASS-MDD: Predictive Visual-Audio Alignment Self-Supervision for Multimodal Deepfake Detection
Yu, Y., Liu, X., Ni, R., Yang, S., Zhao, Y., & Kot, A. C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3309899 · Cited by 63
Diffusion Facial Forgery Detection
Cheng, H., Guo, Y., Wang, T., Nie, L., & Kankanhalli, M. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680797 · Cited by 47
Self-citationIntegrating Audio-Visual Features For Multimodal Deepfake Detection
Muppalla, S., Jia, S., & Lyu, S. (2023) · 2023 IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc60662.2023.10534969 · Cited by 46
“Another branch of methods leverages multimodal detectors that extract the audio-visual inconsistencies in deepfakes [14, 21, 12, 22].”
BackgroundMethodAudio–visual deepfake detection using articulatory representation learning
Wang, Y., & Huang, H. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104133 · Cited by 28
Fine-Grained Multimodal DeepFake Classification via Heterogeneous Graphs
Yin, Q., Lu, W., Cao, X., Luo, X., Zhou, Y., & Huang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02128-1 · Cited by 23
“Besides, Cheng et al. (2023) used a self-supervised manner to learn the audio-visual synchronization patterns in real videos and applied these pretrained feature to detect multimodal forgery.”
"Better Be Computer or I'm Dumb": A Large-Scale Evaluation of Humans as Audio Deepfake Detectors
Warren, K., Tucker, T., Crowder, A., Olszewski, D., Lu, A., Fedele, C., Pasternak, M., Layton, S., Butler, K., Gates, C., et al. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670325 · Cited by 15
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Katamneni, V. S., & Rattani, A. (2024) · 2024 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744480 · Cited by 20
From Black Boxes to Glass Boxes: Explainable AI for Trustworthy Deepfake Forensics
Qian, H., Xia, L., Ge, R., Fan, Y., Wang, Q., & Jing, Z. (2025) · Cryptography · doi:10.3390/cryptography9040061 · Cited by 10
“The Voice-Face Homogeneity approach [33], for example, operates on the premise that an individual’s voice and face share a biometric link.”
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6
Anshul, A., Gopal, S., Rajan, D., & Chng, E. S. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01283 · Cited by 1
Metadata
- Authors
- Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L.
- Venue
- ACM Transactions on Multimedia Computing, Communications, and Applications
- Published
- 2023-11-10
- Volume
- 20
- Issue
- 3
- Pages
- 1-22
- ISSN
- 1551-6857 (ISSN-L)
- 1551-6865
- OpenAlex
- W4386928847
- Cited by
- 102
- References
- 17
- FWCI
- 9.18
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.