Head2Head: Video-based Neural Head Synthesis
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
In this paper, we propose a novel machine learning architecture for facial reenactment. In particular, contrary to the model-based approaches or recent frame-based methods that use Deep Convolutional Neural Networks (DCNNs) to generate individual frames, we propose a novel method that (a) exploits the special structure of facial motion (paying particular attention to mouth motion) and (b) enforces temporal consistency. We demonstrate that the proposed method can transfer facial expressions, pose and gaze of a source actor to a target video in a photo-realistic fashion more accurately than state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
“Least squares generative adversarial networks,”
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.304 · Cited by 5,437
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00917 · Cited by 5,360
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 3×
“To assess the realism of our synthesised videos automatically, we utilise the trained forgery-detection network provided by the authors of [49].”
BackgroundMethodInfluentialFace2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687
Coleman, T. F., & Li, Y. (1996) · SIAM Journal on Optimization · doi:10.1137/s1052623494240456 · Cited by 604 · cited 1×
“Assuming that the camera parameters (P) in (2) have been estimated in an initialisation stage, the minimisation of the cost function results in a large-scale least squares problem with box constraints, which we solve efficiently by using the reflective Newton method of [37].”
MethodWang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594
Research works printed but not linked: 43
- 10.1007/s11263-017-1009-7 · DOI · Paper page
- 10.1109/CVPR.2016.598 · DOI · Paper page
- denton, 2017, Unsupervised learning of disentangled representations from video, NIPS
- 10.1109/ICCV.2017.308 · DOI · Paper page
- taigman, 2016, Unsupervised cross-domain image generation, arXiv 1611 02200
- deng, 2018, The menpo benchmark for multi-pose 2d and 3d facial landmark localisation and tracking, IJCV
- 10.1109/ICCV.2017.126 · DOI · Paper page
- 10.1109/ICCV.2017.244 · DOI · Paper page
- liu, 2017, Unsupervised image-to-image translation networks, arXiv 1703 00848
- 10.1109/ICCV.2017.580 · DOI · Paper page
- 10.1145/383259.383289 · DOI · Paper page
- li, 2017, Learning a model of facial shape and expression from 4d scans, ACM TOG, 10.1145/3130800.3130813 · DOI · Paper page
- thies, 2019, Deferred neural rendering: Image synthesis using neural textures, ACM TOG 2019
- 10.1109/CVPR.2014.537 · DOI · Paper page
- karras, 2018, Progressive growing of GANs for improved quality, stability, and variation, ICLRE
- goodfellow, 2014, Generative adversarial nets, NIPS
- isola, 2016, Image-to-image translation with conditional adversarial networks, ArXiv
- mirza, 2014, Conditional generative adversarial nets, arXiv 1411 1784
- 2019, Faceswap
- 2019, Deepfake
- barron, 2014, Shape, illumination, and reflectance from shading, TPAMI
- 10.1109/ICCVW.2017.16 · DOI
- 10.1109/CVPR.2015.7298604 · DOI
- 10.1109/CVPR.2014.139 · DOI
- 10.1109/CVPR.2013.168 · DOI · Paper page
- grahof, 2017, Projective structure from facial motion, IAPR MVA
- hernandez, 2017, Accurate 3d face reconstruction via prior constrained structure from motion, Computers & Graphics, 10.1016/j.cag.2017.05.008 · DOI
- 10.1145/311535.311556 · DOI · Paper page
- koujan, 2018, Combining dense nonrigid structure from motion and 3d morphable models for monocular 4d face reconstruction, ACM SIGGRAPH CVMP
- 10.1109/TPAMI.2018.2832138 · DOI
- thies, 2015, Real-time expression transfer for facial reenactment, ACM TOG, 10.1145/2816795.2818056 · DOI · Paper page
- 10.1145/1073204.1073209 · DOI · Paper page
- 10.1145/2929464.2929475 · DOI · Paper page
- kim, 2018, Deep video portraits, ACM TOG 2018
- wiles, 2018, X2face: A network for controlling face generation by using images, audio, and pose codes, ECCV
- averbuch-elor, 0, Bringing portraits to life, ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2013
- simonyan, 2015, Very deep convolutional networks for large-scale image recognition, ICLRE
- 10.1109/CVPR.2018.00537 · DOI
- 10.1109/CVPR.2017.179 · DOI · Paper page
- 10.1016/j.imavis.2016.10.007 · DOI
- cao, 2014, Facewarehouse: A 3d facial expression database for visual computing, TVCG
- guo, 2018, Stacked dense u-nets with dual transformers for robust face alignment, arXiv preprint arXiv 1812 08942
- park, 2018, Deep pictorial gaze estimation, ECCV
Cited by
Multi-attentional Deepfake Detection
Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00222 · Cited by 1,019
“Benefiting from the great progress in generative models, deepfake techniques have achieved significant success recently and various face forgery methods [19, 41, 21, 31, 32, 44, 28, 38] have been proposed.”
BackgroundM2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N. (2022) · Proceedings of the 2022 International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531415 · Cited by 368
“Recent years have witnessed the rapid development of Deepfake techniques [28, 31, 45, 54], which enable attackers to manipulate”
BackgroundHeadGAN: One-shot Neural Head Synthesis and Editing
Doukas, M. C., Zafeiriou, S., & Sharmanska, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01413 · Cited by 117
“Deep video portraits (DVP) [23] and Head2Head [25] are examples of such reenactment systems driven by 3D information.”
BackgroundSelf-citationDelving into the Local: Dynamic Inconsistency Learning for DeepFake Video Detection
Gu, Z., Chen, Y., Yao, T., Ding, S., Li, J., & Ma, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19955 · Cited by 93
“With the rapid development of deep learning-based meth-ods, especially generative models, various DeepFake techniques (Koujan et al. 2020; Nirkin, Keller, and Hassner 2019) have been proposed.”
Method
Metadata
- Authors
- Koujan, M. R., Doukas, M. C., Roussos, A., & Zafeiriou, S.
- Venue
- 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)
- Publisher
- IEEE
- Published
- 2020-11
- Pages
- 16-23
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W3027351130
- Cited by
- 61
- References
- 9
- FWCI
- 3.57
- Type
- conference paper
- Language
- en
- Open access
- green
- Licence
- Funders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.