Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 475

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Deep video portraits

Kim, Hyeongwoo, Garrido, Pablo, Tewari, Ayush, Xu, Weipeng, Thies, Justus, Niessner, Matthias, Pérez, Patrick, Richardt, Christian, Zollhöfer, Michael, Theobalt, Christian·2018·ACM Transactions on Graphics
DOI 10.1145/3197517.3201283
Open this paper in CoChat — free account

Abstract

We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network - thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3197517.3201283

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Input
InputCC-BY · From the PDF
Output
OutputCC-BY · From the PDF
Image 5
CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Fig. 2 Deep video portraits enable a source actor to fully control a target video portrait. First, a low-dimensional parametric representation (let) of both videos is obtained using monocular face reconstruction. The head pose, expression and eye gaze can now be transferred in parameter space (middle). We do not focus on the modification of the identity and scene illumination (hatched background), since we are interested in reenactment. Finally, we render conditioning input images that are converted to a photo-realistic video portrait of the target actor (right). Obama video courtesy of the White House (public domain).
Fig. 2 Deep video portraits enable a source actor to fully control a target video portrait. First, a low-dimensional parametric representation (let) of both videos is obtained using monocular face reconstruction. The head pose, expression and eye gaze can now be transferred in parameter space (middle). We do not focus on the modification of the identity and scene illumination (hatched background), since we are interested in reenactment. Finally, we render conditioning input images that are converted to a photo-realistic video portrait of the target actor (right). Obama video courtesy of the White House (public domain).CC-BY · From the PDF
Fig. 3 The synthetic input used for conditioning our rendering-to-video translation network: (1) colored face rendering under target illumination, (2) correspondence image, and (3) the eye gaze image.
Fig. 3 The synthetic input used for conditioning our rendering-to-video translation network: (1) colored face rendering under target illumination, (2) correspondence image, and (3) the eye gaze image.CC-BY · From the PDF
Fig. 4 Architecture of our rendering-to-video translation network for an input resolution of 256 × 256 : The encoder has 8 downsampling modules with ( 64 , 128 , 256 , 512 , 512 , 512 , 512 , 512 ) output channels. The decoder has 8 upsampling modules with ( 512 , 512 , 512 , 512 , 256 , 128 , 64 , 3 ) output channels. The upsampling modules use the following dropout probabilities ( 0 . 5 , 0 . 5 , 0 . 5 , 0 , 0 , 0 , 0 , 0 ) . The first downsampling and the last upsampling module do not employ batch normalization (BN). The final non-linearity (TanH) brings the output to the employed normalized [-1 , + 1 ] -space.
Fig. 4 Architecture of our rendering-to-video translation network for an input resolution of 256 × 256 : The encoder has 8 downsampling modules with ( 64 , 128 , 256 , 512 , 512 , 512 , 512 , 512 ) output channels. The decoder has 8 upsampling modules with ( 512 , 512 , 512 , 512 , 256 , 128 , 64 , 3 ) output channels. The upsampling modules use the following dropout probabilities ( 0 . 5 , 0 . 5 , 0 . 5 , 0 , 0 , 0 , 0 , 0 ) . The first downsampling and the last upsampling module do not employ batch normalization (BN). The final non-linearity (TanH) brings the output to the employed normalized [-1 , + 1 ] -space.CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Image 14
CC-BY · From the PDF
Rotation
RotationCC-BY · From the PDF
Image 16
CC-BY · From the PDF
Reference
ReferenceCC-BY · From the PDF
Expression
ExpressionCC-BY · From the PDF
Fig. 10 Interactive editing. Our approach provides full parametric control over video portraits (by controlling head model parameters in conditioning images). This enables modifications of the rigid head pose (rotation and translation), facial expression and eye motion. All of these dimensions can be manipulated together or independently. We also show these modifications live in the supplemental video. Obama video courtesy of the White House (public domain).
Fig. 10 Interactive editing. Our approach provides full parametric control over video portraits (by controlling head model parameters in conditioning images). This enables modifications of the rigid head pose (rotation and translation), facial expression and eye motion. All of these dimensions can be manipulated together or independently. We also show these modifications live in the supplemental video. Obama video courtesy of the White House (public domain).CC-BY · From the PDF
Identity Change
Identity ChangeCC-BY · From the PDF
Fig. 12 Comparison to the image reenactment approach of Averbuch-Elor et al. [2017] in the full-head reenactment scenario. Since their method is based on a single target image, they copy the mouth interior from the source to the target, thus not preserving the target's identity. Our learningbased approach enables larger modifications of the rigid head pose without apparent artifacts, while their warping-based approach distorts the head and background. In addition, ours enables joint control of the eye gaze and eye blinks. The diferences are most evident in the supplemental video. Obama video courtesy of the White House (public domain).
Fig. 12 Comparison to the image reenactment approach of Averbuch-Elor et al. [2017] in the full-head reenactment scenario. Since their method is based on a single target image, they copy the mouth interior from the source to the target, thus not preserving the target's identity. Our learningbased approach enables larger modifications of the rigid head pose without apparent artifacts, while their warping-based approach distorts the head and background. In addition, ours enables joint control of the eye gaze and eye blinks. The diferences are most evident in the supplemental video. Obama video courtesy of the White House (public domain).CC-BY · From the PDF
Fig. 13 Comparison to Suwajanakorn et al. [2017]. Their approach produces accurate lip sync with visually imperceptible artifacts, but provides no direct control over facial expressions. Thus, the expressions in the output do not always perfectly match the input (box, mouth), especially for expression changes without audio cue. Our visual dubbing approach accurately transfers the expressions from the source to the target. In addition, our approach provides more control over the target video by also transferring the eye gaze and eye blinks (box, eyes), and the rigid head pose (arrows). Since the source sequence shows more head-pose variation than the target sequence, we scaled the transferred rotation and translation by 0.5 in this experiment. For the full video sequence, we refer to the supplemental video. Obama video courtesy of the White House (public domain).
Fig. 13 Comparison to Suwajanakorn et al. [2017]. Their approach produces accurate lip sync with visually imperceptible artifacts, but provides no direct control over facial expressions. Thus, the expressions in the output do not always perfectly match the input (box, mouth), especially for expression changes without audio cue. Our visual dubbing approach accurately transfers the expressions from the source to the target. In addition, our approach provides more control over the target video by also transferring the eye gaze and eye blinks (box, eyes), and the rigid head pose (arrows). Since the source sequence shows more head-pose variation than the target sequence, we scaled the transferred rotation and translation by 0.5 in this experiment. For the full video sequence, we refer to the supplemental video. Obama video courtesy of the White House (public domain).CC-BY · From the PDF
Image 23
CC-BY · From the PDF
500images
500imagesCC-BY · From the PDF
500 images
500 imagesCC-BY · From the PDF
Fig. 15 uantitative evaluation of the training set size. We train our rendering-to-video translation network with training corpora of diferent sizes. The error maps show the per-pixel distance in RGB color space with each channel being in [ 0 , 255 ] ; the mean photometric error is shown in the top-right. Smaller training sets have larger photometric errors, especially for regions outside of the face. For the full comparison, we refer to the supplemental video. Obama video courtesy of the White House (public domain). May video courtesy of the UK government (Open Government Licence).
Fig. 15 uantitative evaluation of the training set size. We train our rendering-to-video translation network with training corpora of diferent sizes. The error maps show the per-pixel distance in RGB color space with each channel being in [ 0 , 255 ] ; the mean photometric error is shown in the top-right. Smaller training sets have larger photometric errors, especially for regions outside of the face. For the full comparison, we refer to the supplemental video. Obama video courtesy of the White House (public domain). May video courtesy of the UK government (Open Government Licence).CC-BY · From the PDF
Output Error Single Frame
Output Error Single FrameCC-BY · From the PDF
Output Error Space-time Conditioning
Output Error Space-time ConditioningCC-BY · From the PDF
256× 256
256× 256CC-BY · From the PDF
Fig. 18 Weperformed a user study to evaluate the quality of our results and see if users can distinguish between real (top) and synthesized video clips (botom). The video clips include self-reenactment, same-person-reenactment, and video dubbing. Putin video courtesy of the Kremlin (CC BY). Obama video courtesy of the White House (public domain). Elizabeth II video courtesy of the Governor General of Canada (public domain).
Fig. 18 Weperformed a user study to evaluate the quality of our results and see if users can distinguish between real (top) and synthesized video clips (botom). The video clips include self-reenactment, same-person-reenactment, and video dubbing. Putin video courtesy of the Kremlin (CC BY). Obama video courtesy of the White House (public domain). Elizabeth II video courtesy of the Governor General of Canada (public domain).CC-BY · From the PDF
Fig. 19 Our approach works well within the span of the training corpus. Extreme changes in head pose far outside the training set or strong changes to the facial expression might lead to artifacts in the synthesized video. This is a common limitation of all learning-based approaches. In these cases, artifacts are most prominent outside the face region, as these regions have no conditioning input. May video courtesy of the UK government (Open Government Licence). Malou video courtesy of Louisa Malou (CC BY).
Fig. 19 Our approach works well within the span of the training corpus. Extreme changes in head pose far outside the training set or strong changes to the facial expression might lead to artifacts in the synthesized video. This is a common limitation of all learning-based approaches. In these cases, artifacts are most prominent outside the face region, as these regions have no conditioning input. May video courtesy of the UK government (Open Government Licence). Malou video courtesy of Louisa Malou (CC BY).CC-BY · From the PDF
Image 32
CC-BY · From the PDF
Malou 15,000
Malou 15,000CC-BY · From the PDF
May
MayCC-BY · From the PDF
5,000
5,000CC-BY · From the PDF
Obama1 2,000
Obama1 2,000CC-BY · From the PDF
Image 37
CC-BY · From the PDF
Image 38
CC-BY · From the PDF
Obama2 3,613
Obama2 3,613CC-BY · From the PDF
Reagan 6,984
Reagan 6,984CC-BY · From the PDF
Thomas 2,239
Thomas 2,239CC-BY · From the PDF
DB3 6,500
DB3 6,500CC-BY · From the PDF
DB1 8,000
DB1 8,000CC-BY · From the PDF
DB4 30,024
DB4 30,024CC-BY · From the PDF
Obama3 1,945
Obama3 1,945CC-BY · From the PDF
Image 46
CC-BY · From the PDF
David1 4,611
David1 4,611CC-BY · From the PDF
David2 3,323
David2 3,323CC-BY · From the PDF
DB6 2,380
DB6 2,380CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. Ian J. Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems. Ian J. Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems.
  2. Ira Kemelmacher-Shlizerman , Aditya Sankar , Eli Shechtman , and Steven M. Seitz . 2010 . Being John Malkovich. In European Conference on Computer Vision (ECCV). 341--353 . Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman, and Steven M. Seitz. 2010. Being John Malkovich. In European Conference on Computer Vision (ECCV). 341--353.

Cited by

See all 475

Metadata

Authors
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C.
Affiliations
  • Max Planck Institute for Informatics, Germany
  • Technicolor, France
  • Technical University of Munich, Germany
  • University of Bath, United Kingdom
  • Stanford University
Venue
ACM Transactions on Graphics
Publisher
Association for Computing Machinery (ACM)
Published
2018-07-30
Volume
37
Issue
4
Pages
1-14
ISSN
  • 0730-0301
  • 1557-7368
DOI
10.1145/3197517.3201283
Cited by
475
References
74
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Funders
  • ERC Starting Grant CapReal (335545)
  • Google Faculty Award
  • RCUK grant CAMERA (EP/M023281/1)
  • Max Planck Center for Visual Computing and Communications
  • TUM-IAS Rudolf Mößbauer Fellowship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.