CoChat
Sign Up
Cited by 181Digital Media Forensic DetectionFace recognition and analysisGenerative Adversarial Networks and Image Synthesis

Protecting Celebrities from DeepFake with Identity Consistency Transformer

Open access
Dong, Xiaoyi, Bao, Jianmin, Chen, Dongdong, Zhang, Ting, Zhang, Weiming, Yu, Nenghai, Chen, Dong, Wen, Fang +1 more·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.00925
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work we propose Identity Consistency Transformer, a novel face forgery detection method that focuses on high-level semantics, specifically identity information, and detecting a suspect face by finding identity inconsistency in inner and outer face regions. The Identity Consistency Transformer incorporates a consistency loss for identity consistency determination. We show that Identity Consistency Transformer exhibits superior generalization ability not only across different datasets but also across various types of image degradation forms found in real-world applications including deepfake videos. The Identity Consistency Transformer can be easily enhanced with additional identity information when such information is available, and for this reason it is especially well-suited for detecting face forgeries involving celebrities.11Code will be released at https://github.com/LightDXY/ICT_DeepFake

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.00925

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Characterizing pristine faces based on semantic features such as soft biometrics is particularly effective for POI-based detection.

    “This approach turns out to be particularly effective if the characterization of pristine faces is based on semantic features, such as soft biometrics, leading to Person-of-Interest (POI) based detection [3,5-7,12,20].”

    From Audio-Visual Person-of-Interest DeepFake Detection (2023) · cites this paper as [20]

    The Introduction and §3.2 describe using high-level identity information from authentic celebrity face images to improve reference-assisted forgery detection. The paper does not characterize those features as soft biometrics; its method extracts inner- and outer-face identity vectors and tests their consistency. Thus it supports the semantic, identity-based detection point, but not the specific soft-biometrics example. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Jawadul H Bappy, Cody Simons, Lakshmanan Nataraj, BS Manjunath, and Amit K Roy-Chowdhury. Hybrid lstm and encoder-decoder architecture for detection of image forg- eries. IEEE Transactions on Image Processing, 28(7):3286- 3300, 2019. 1, 2
  2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3 · arXiv
  3. Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413-425, 2013. 2
  4. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In International Conference on Ma- chine Learning, pages 1691-1703. PMLR, 2020. 3
  5. Yi-Ting Cheng, Virginia Tzeng, Yu Liang, Chuan-Chang Wang, Bing-Yu Chen, Yung-Yu Chuang, and Ming Ouhy- oung. 3d-model-based face replacement in video. In SIG- GRAPH'09: Posters, page 29. ACM, 2009. 2
  6. Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, and Luisa Verdoliva. Id-reveal: Identity-aware deepfake video detection. arXiv preprint arXiv:2012.02512, 2020. 3 · arXiv
  7. Kevin Dale, Kalyan Sunkavalli, Micah K Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. Video face replacement. In ACM Transactions on Graphics (TOG), vol- ume 30, page 130. ACM, 2011. 2
  8. Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5781-5790, 2020. 6
  9. Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Bain- ing Guo. Cswin transformer: A general vision trans- former backbone with cross-shaped windows. arXiv preprint arXiv:2107.00652, 2021. 3 · arXiv
  10. Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9821-9830, 2019. 2
  11. Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pages 87-102. Springer, 2016. 4, 5
  12. Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 3677-3685, 2017. 2
  13. Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Faceshifter: Towards high fidelity and occlusion aware face swapping. arXiv preprint arXiv:1912.13457, 2019. 1, 2 · arXiv
  14. Yuan Lin, Shengjin Wang, Qian Lin, and Feng Tang. Face swapping under large pose variations: A 3d model based ap- proach. In 2012 IEEE International Conference on Multime- dia and Expo, pages 333-338. IEEE, 2012. 2
  15. Yaqi Liu, Qingxiao Guan, Xianfeng Zhao, and Yun Cao. Im- age forgery localization based on multi-scale convolutional neural networks. In Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia Security, pages 85- 90, 2018. 1, 2
  16. Iacopo Masi, Aditya Killekar, Royston Marian Mascaren- has, Shenoy Pratik Gurudatt, and Wael AbdAlmageed. Two- branch recurrent network for isolating deepfakes in videos. In European Conference on Computer Vision, pages 667- 684. Springer, 2020. 6
  17. Iacopo Masi, Anh Tuan Tran, Tal Hassner, Jatuporn Toy Lek- sut, and Gerard Medioni. Do we really need to collect mil- lions of faces for effective face recognition? In European conference on computer vision, pages 579-596. Springer, 2016. 2
  18. Iacopo Masi, Anh Tuan Tran, Tal Hassner, Gozde Sahin, and Gerard Medioni. Face-specific data augmentation for uncon- strained face recognition. International Journal of Computer Vision, 127(6):642-667, 2019. 2
  19. Falko Matern, Christian Riess, and Marc Stamminger. Ex- ploiting visual artifacts to expose deepfakes and face manip- ulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pages 83-92. IEEE, 2019. 1, 2
  20. Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. Rsgan: face swapping and editing using face and hair repre- sentation in latent spaces. arXiv preprint arXiv:1804.03447, 2018. 2 · arXiv
  21. Yuval Nirkin, Iacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni. On face segmentation, face swapping, and face perception. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 98-105. IEEE, 2018. 1, 2
  22. Yuval Nirkin, Lior Wolf, Yosi Keller, and Tal Hassner. Deep- fake detection based on the discrepancy between the face and its context. arXiv preprint arXiv:2008.12262, 2020. 3, 6 · arXiv
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 3
  24. Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017. 2
  25. Justus Thies, Michael Zollhöfer, and Matthias Nießner. De- ferred neural rendering: Image synthesis using neural tex- tures. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019. 5
  26. Justus Thies, Michael Zollhofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2387-2395, 2016. 2, 5
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Il- lia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. 3, 4 · arXiv
  28. Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transform- ers. In ICCV, 2021. 3
  29. Hong-Xia Wang, Chunhong Pan, Haifeng Gong, and Huai- Yu Wu. Facial image composition based on active appear- ance model. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 893-896. IEEE, 2008. 2
  30. Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. arXiv preprint arXiv:2112.01529, 2021. 3 · arXiv
  31. Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are sur- prisingly easy to spot... for now. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 7, 2020. 6
  32. Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deep- fake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 2
  33. Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuan- jun Xiong, and Wei Xia. Learning to recognize patch- wise consistency for deepfake detection. arXiv preprint arXiv:2012.09311, 2020. 6 · arXiv
  34. Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. Exploring temporal coherence for more gen- eral video face forgery detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15044-15054, October 2021. 2

Cited by

Sorted by citations

Authors
Dong, X., Bao, J., Chen, D., Zhang, T., Zhang, W., Yu, N., Chen, D., Wen, F., & Guo, B.
Affiliations
  • University of Science and Technology of China (CN)
  • Microsoft Research Asia (China) (CN)
ORCID
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
9458-9468
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.00925
OpenAlex
W4225966148
Cited by
181
References
9
Citations by year
2026: 27 · 2025: 52 · 2024: 60 · 2023: 38 · 2022: 4
FWCI
13.96
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Natural Science Foundation of China (U20B2047)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deepfake detection, face forgery detection, celebrity image, identity consistency, consistency loss, identity information, image degradation robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review