CoChat
Sign Up
PreprintCited by 236Generative Adversarial Networks and Image SynthesisFace recognition and analysisDigital Media Forensic Detection

FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

Open access
Li, Lingzhi, Bao, Jianmin, Yang, Hao, Chen, Dong, Wen, Fang·2019·arXiv
DOI 10.48550/arxiv.1912.13457arXiv 1912.13457
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work, we propose a novel two-stage framework, called FaceShifter, for high fidelity and occlusion aware face swapping. Unlike many existing face swapping works that leverage only limited information from the target image when synthesizing the swapped face, our framework, in its first stage, generates the swapped face in high-fidelity by exploiting and integrating the target attributes thoroughly and adaptively. We propose a novel attributes encoder for extracting multi-level target face attributes, and a new generator with carefully designed Adaptive Attentional Denormalization (AAD) layers to adaptively integrate the identity and the attributes for face synthesis. To address the challenging facial occlusions, we append a second stage consisting of a novel Heuristic Error Acknowledging Refinement Network (HEAR-Net). It is trained to recover anomaly regions in a self-supervised way without any manual annotations. Extensive experiments on wild faces demonstrate that our face swapping results are not only considerably more perceptually appealing, but also better identity preserving in comparison to other state-of-the-art methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1912.13457

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Videos created in DeeperForensics-1.0 use deep fake generation techniques similar to FaceForensics++.

    “Whereas, the videos created in DeeperForensics-1.0 uses deep fake generation techniques similar to FaceForenics++.”

    From On Improving Cross-dataset Generalization of Deepfake Detectors (2022) · cites this paper as [13]

    The comparison section (pp. 5–6) discusses face-swapping methods tested on FaceForensics++ images and videos, but does not address how DeeperForensics-1.0 videos were generated. Searches for “DeeperForensics-1.0” and “DeeperForensics” found no passage establishing the claimed similarity. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [3][3] Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE International Conference on Computer Vision , pages 1949-1957, 2015. 5, 11 -
  2. [4][4] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan: fine-grained image generation through asymmetric training. In Proceedings of the IEEE International Conference on Computer Vision , pages 2745-2754, 2017. 1 -
  3. [5][5] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Towards open-set identity preserving face synthesis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018. 2, 3, 5, 6, 7 -
  4. [7][7] Volker Blanz, Kristina Scherbaum, Thomas Vetter, and Hans-Peter Seidel. Exchanging faces in images. In Computer Graphics Forum , volume 23, pages 669-676. Wiley Online Library, 2004. 1, 2, 3 -
  5. [8][8] Volker Blanz, Thomas Vetter, et al. A morphable model for the synthesis of 3d faces. In Siggraph , volume 99, pages 187-194, 1999. 3 -
  6. [9][9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015. 5, 11 - · arXiv
  7. [10][10] Bindita Chaudhuri, Noranart Vesdapunt, and Baoyuan Wang. Joint face detection and facial motion retargeting for multiple faces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9719-9728, 2019. 6 -
  8. [11][11] Dong Chen, Shaoqing Ren, Yichen Wei, Xudong Cao, and Jian Sun. Joint cascade face detection and alignment. In European Conference on Computer Vision , pages 109-122. Springer, 2014. 5 -
  9. [12][12] Yi-Ting Cheng, Virginia Tzeng, Yu Liang, Chuan-Chang Wang, Bing-Yu Chen, Yung-Yu Chuang, and Ming Ouhyoung. 3d-model-based face replacement in video. In SIGGRAPH'09: Posters , page 29. ACM, 2009. 1 -
  10. [14][14] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 , 2016. 3 - · arXiv
  11. [16][16] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision , pages 1501-1510, 2017. 3 -
  12. [17][17] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. 3 - · arXiv
  13. [18][18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017. 5 - · arXiv
  14. [20][20] Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick P´ erez, Christian Richardt, Michael Zollh¨ ofer, and Christian Theobalt. Deep video portraits. ACMTransactions on Graphics (TOG) , 37(4):163, 2018. 2 -
  15. [22][22] Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 3677-3685, 2017. 1, 2 -
  16. [23][23] Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 3677-3685, 2017. 3 -
  17. [24][24] Yin Li, Alireza Fathi, and James M Rehg. Learning to predict gaze in egocentric video. In Proceedings of the IEEE International Conference on Computer Vision , pages 3216-3223, 2013. 5, 11 -
  18. [25][25] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 287-295, 2015. 5, 11 -
  19. [26][26] Yuan Lin, Shengjin Wang, Qian Lin, and Feng Tang. Face swapping under large pose variations: A 3d model based approach. In 2012 IEEE International Conference on Multimedia and Expo , pages 333-338. IEEE, 2012. 1 -
  20. [27][27] Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li. pagan: real-time avatars using dynamic textures. In SIGGRAPH Asia 2018 Technical Papers , page 258. ACM, 2018. 2, 3 -
  21. [28][28] Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. Fsnet: An identity-aware generative model for image-based face swapping. In Asian Conference on Computer Vision , pages 117-132. Springer, 2018. 1, 2, 3 -
  22. [29][29] Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. Rsgan: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 , 2018. 1, 2, 3 - · arXiv
  23. [30][30] Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE International Conference on Computer Vision , pages 7184-7193, 2019. 1, 2, 3, 5, 6 -
  24. [31][31] Yuval Nirkin, Iacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni. On face segmentation, face swapping, and face perception. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) , pages 98-105. IEEE, 2018. 1, 2, 5, 6, 7 -
  25. [32][32] Kyle Olszewski, Zimo Li, Chao Yang, Yi Zhou, Ronald Yu, Zeng Huang, Sitao Xiang, Shunsuke Saito, Pushmeet Kohli, and Hao Li. Realistic dynamic facial textures from a single image using gans. In Proceedings of the IEEE International Conference on Computer Vision , pages 5429-5438, 2017. 2, 3 -
  26. [33][33] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2337-2346, 2019. 3, 4, 11 -
  27. [35][35] Arun Ross and Asem Othman. Visual cryptography for biometric privacy. IEEE transactions on information forensics and security , 6(1):70-81, 2010. 1 -
  28. [36][36] Andreas R¨ ossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. arXiv preprint arXiv:1901.08971 , 2019. 2, 5, 6 - · arXiv
  29. [37][37] Nataniel Ruiz, Eunji Chong, and James M Rehg. Finegrained head pose estimation without keypoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 2074-2083, 2018. 6 -
  30. [38][38] Justus Thies, Michael Zollh¨ ofer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. arXiv preprint arXiv:1904.12356 , 2019. 2 - · arXiv
  31. [39][39] Justus Thies, Michael Zollh¨ ofer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. Realtime expression transfer for facial reenactment. ACM Trans. Graph. , 34(6):183-1, 2015. 2 -
  32. [40][40] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2387-2395, 2016. 2 -
  33. [42][42] Hong-Xia Wang, Chunhong Pan, Haifeng Gong, and HuaiYu Wu. Facial image composition based on active appearance model. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 893-896. IEEE, 2008. 1, 2 3 × 256 × 256 AAD ResBlk (64,3) 64 × 256 × 256 BilinearUpsample x2 AAD ResBlk (128,64) 128 × 128 × 128 BilinearUpsample x2 AAD ResBlk (256,128) 256 × 64 × 64 BilinearUpsample x2 AAD ResBlk (512,256) 512 × 32 × 32 BilinearUpsample x2 AAD ResBlk (1024,512) 1024 × 16 × 16 BilinearUpsample x2 AAD ResBlk (1024,1024) 1024 × 8 × 8 BilinearUpsample x2 AAD ResBlk (1024,1024) 1024 × 4 × 4 BilinearUpsample x2 AAD ResBlk (1024,1024) 1024 × 2 × 2 nvTranspose 2 × 2 -Generator ,1,0 ෠ 𝑌 𝑠,𝑡

Cited by

Sorted by citations

Authors
Li, L., Bao, J., Yang, H., Chen, D., & Wen, F.
Affiliations
  • Peking University (CN)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1912.13457
arXiv
1912.13457
OpenAlex
W2998158928
Cited by
236
References
6
Citations by year
2025: 35 · 2024: 48 · 2023: 60 · 2022: 34 · 2021: 45 · 2020: 14
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
face swapping, face occlusion, identity preservation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review