CoChat
Sign Up
Cited by 101Face recognition and analysisGenerative Adversarial Networks and Image Synthesis3D Shape Modeling and Analysis

LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization

Open access
Lahiri, Avisek, Kwatra, Vivek, Frueh, Christian, Lewis, John, Bregler, Chris·2021·2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr46437.2021.00278
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we present a video-based learning framework for animating personalized 3D talking faces from audio. We introduce two training-time data normalizations that significantly improve data sample efficiency. First, we isolate and represent faces in a normalized space that decouples 3D geometry, head pose, and texture. This decomposes the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. Second, we leverage facial symmetry and approximate albedo constancy of skin to isolate and remove spatio-temporal lighting variations. Together, these normalizations allow simple networks to generate high fidelity lip-sync videos under novel ambient illumination while training with just a single speaker-specific video. Further, to stabilize temporal dynamics, we introduce an auto-regressive approach that conditions the model on its previous visual state. Human ratings and objective metrics demonstrate that our method outperforms contemporary state-of-the-art audio-driven video reenactment benchmarks in terms of realism, lip-sync and visual quality scores. We illustrate several applications enabled by our framework.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr46437.2021.00278

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Methods that concentrate on partial facial attributes such as lip movements or emotional biases are easily countered by specific countermeasures.

    “These methods, though improved generalization, however, concentrate on partial facial attributes, such as lip movements or emotional biases, which are easily countered by specific countermeasures [44, 45].”

    From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [45]

    The Sharpness of Synthesized Lip Region section evaluates the method’s lower-mouth output, supporting the characterization of this work as focused on lip movements. The cited paper does not say that such methods focus on emotional biases or are easily defeated by specific countermeasures; its Ethical Considerations section discusses fake-content detection as a research need, not a demonstrated countermeasure. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. Lipnet: End- to-end sentence-level lipreading. arXiv preprint arXiv:1611.01599, 2016. 7 · arXiv
  2. Thabo Beeler and Derek Bradley. Rigid stabilization of facial expressions. ACM Trans. Graph., 33(4), July 2014. 3
  3. J. Booth and S. Zafeiriou. Optimal UV spaces for facial morphable model construction. In IEEE ICIP, pages 4672-4676, 2014. 3
  4. Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. In ECCV, pages 520-535, 2018. 1, 2, 8
  5. Lele Chen, Ross K Maddox, Zhiyao Duan, and Chen- liang Xu. Hierarchical cross-modal talking face gen- eration with dynamic pixel-wise loss. In CVPR, pages 7832-7841, 2019. 1, 2, 7, 8
  6. Joon Son Chung, Amir Jamaludin, and Andrew Zis- serman. You said that? In BMVC, 2017. 2, 8
  7. Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech percep- tion and automatic speech recognition. The Journal of the Acoustical Society of America, 120(5):2421-2424, November 2006. 7, 14
  8. Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learn- ing, and synthesis of 3D speaking styles. In CVPR, pages 10101-10111, 2019. 1
  9. Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learn- ing, and synthesis of 3d speaking styles. In CVPR, pages 10101-10111, 2019. 2
  10. Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. JALI: an animator-centric viseme model for expressive lip synchronization. ACM TOG, 35(4):127, 2016. 1, 2
  11. Awni Hannun, Carl Case, Jared Casper, Bryan Catan- zaro, Greg Diamos, Erich Elsen, Ryan Prenger, San- jeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recogni- tion. arXiv preprint arXiv:1412.5567, 2014. 2 · arXiv
  12. Naomi Harte and Eoin Gillen. TCD-TIMIT: An audio-visual corpus of continuous speech. IEEE Transactions on Multimedia, 17(5):603-615, 2015. 7, 14
  13. Ahmed Hussen Abdelaziz, Barry-John Theobald, Paul Dixon, Reinhard Knothe, Nicholas Apostoloff, and Sachin Kajareker. Modality dropout for improved performance-driven talking faces. In International Conference on Multimodal Interaction, pages 378- 386, 2020. 2
  14. Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM TOG, 36(4):94:1-94:12, July 2017. 1, 2, 5
  15. Yury Kartynnik, Artsiom Ablavatski, Ivan Gr- ishchenko, and Matthias Grundmann. Real-time fa- cial surface geometry from monocular video on mo- bile GPUs. In Third Workshop on Computer Vision for AR/VR, Long Beach, CA, 2019. 2, 3
  16. Michael K Keutmann, Samantha L Moore, Adam Savitt, and Ruben C Gur. Generating an item pool for translational social cognition research: methodol- ogy and initial validation. Behavior research methods, 47(1):228-234, 2015. 7, 14
  17. H. Kim, P. Garrido, A. Tewari, W. Xu, J. Thies, N. Nießner, P. Pérez, C. Richardt, M. Zollhöfer, and C. Theobalt. Deep Video Portraits. ACM TOG, 2018. 3, 8, 16
  18. Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar. Towards automatic face-to-face translation. In ACM Multimedia, pages 1428-1436, 2019. 2, 13
  19. Mathieu Lamarre, J.P. Lewis, and Etienne Danvoye. Face stabilization by mode pursuit for avatar construc- tion. In Image and Vision Computing, pages 1-6. IEEE, 2018. 3
  20. Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM TOG, 36(6):194, 2017. 2
  21. Michael Mathieu, Camille Couprie, and Yann Le- Cun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016. 5
  22. Gaurav Mittal and Baoyuan Wang. Animating face using disentangled audio representations. In WACV, 2019. 1
  23. Linsen Song, Wayne Wu, Chen Qian, Chen Qian, and Chen Change Loy. Everybody's talkin': Let me talk as you want. arXiv preprint, arXiv:, 2020. 3
  24. Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing Obama: learning lip sync from audio. ACM (TOG), 36(4):95, 2017. 1, 3, 13
  25. Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In ECCV, pages 716-731. Springer, 2020. 1, 2, 3, 6, 7, 8, 11
  26. Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In CVPR, pages 2387-2395, 2016. 3, 8, 16
  27. Shinji Umeyama. Least-squares estimation of trans- formation parameters between two point patterns. IEEE TPAMI, 13(4):376-380, 1991. 3
  28. Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. End-to-end speech-driven realistic facial an- imation with temporal gans. In CVPR Workshops, pages 37-40, 2019. 2, 7, 8
  29. Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with GANs. IJCV, pages 1-16, 2019. 2, 7, 8, 11, 14
  30. Lijuan Wang, Wei Han, Frank K Soong, and Qiang Huo. Text driven 3D photo-realistic talking head. In Twelfth Annual Conference of the International Speech Communication Association, 2011. 1, 2
  31. Olivia Wiles, A Sophia Koepke, and Andrew Zisser- man. X2face: A network for controlling face genera- tion using images, audio, and pose codes. In ECCV, pages 670-686, 2018. 1, 2, 8
  32. R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989. 5, 14
  33. Qingxiong Yang, Jinhui Tang, and Narendra Ahuja. Efficient and robust specular highlight removal. IEEE TPAMI, 37(6):1304-1311, 2014. 5
  34. Yang Zhou, Zhan Xu, Chris Landreth, Evange- los Kalogerakis, Subhransu Maji, and Karan Singh. Visemenet: Audio-driven animator-centric speech an- imation. ACM TOG, 37(4):161, 2018.

Cited by

Sorted by citations

Authors
Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C.
Affiliations
Google (United States) (US)
ORCID
Venue
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2021-06
Pages
2754-2763
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr46437.2021.00278
OpenAlex
W3171622771
Cited by
101
References
6
Citations by year
2026: 3 · 2025: 21 · 2024: 27 · 2023: 29 · 2022: 20 · 2021: 1
FWCI
6.18
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
Keywords
lip synchronization, data-efficient learning, pose normalization, facial symmetry

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review