The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
MakeltTalk: speaker-aware talking-head animation
Open accessAbstract
We present a method that generates expressive talking-head videos from a single facial image with audio as the only input. In contrast to previous attempts to learn direct mappings from audio to raw pixels for creating talking faces, our method first disentangles the content and speaker information in the input audio signal. The audio content robustly controls the motion of lips and nearby facial regions, while the speaker information determines the specifics of facial expressions and the rest of the talking-head dynamics. Another key component of our method is the prediction of facial landmarks reflecting the speaker-aware dynamics. Based on this intermediate representation, our method works with many portrait images in a single unified framework, including artistic paintings, sketches, 2D cartoon characters, Japanese mangas, and stylized caricatures. In addition, our method generalizes well for faces and characters that were not observed during training. We present extensive quantitative and qualitative evaluation of our method, in addition to user studies, demonstrating generated talking-heads of significantly higher quality compared to prior state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2008) · Journal of Machine Learning Research · Cited by 33,051
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
“Least squares generative adversarial networks,”
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.304 · Cited by 5,437
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Segal, A., Haehnel, D., & Thrun, S. (2009) · Robotics: Science and Systems V · doi:10.15607/rss.2009.v.021 · Cited by 1,526
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464
Bulat, A., & Tzimiropoulos, G. (2017) · Nottingham ePrints (University of Nottingham) · Cited by 1,277
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
Generalized End-to-End Loss for Speaker Verification
Wan, L., Wang, Q., Papir, A., & Moreno, I. L. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462665 · Cited by 820
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687
GANimation: Anatomically-Aware Facial Animation from a Single Image
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A., & Moreno-Noguer, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_50 · Cited by 621
Demo of Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016) · ACM SIGGRAPH 2016 Emerging Technologies · doi:10.1145/2929464.2929475 · Cited by 589
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585
A Variational U-Net for Conditional Appearance and Shape Generation
Esser, P., & Sutter, E. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00923 · Cited by 454
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437 · cited 4×
“In particular, ourmethod has 2x times lessD-LL error in lip landmark positions compared to [Eskimez et al. 2018], and 2.5x times less D-LL error compared to [Chen et al. 2019].”
BackgroundMethodResultInfluentialSUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit
Veaux, C., Yamagishi, J., & MacDonald, K. (2016) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/1495 · Cited by 434
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416
(2019) · Cited by 415
Brand, M. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311537 · Cited by 411
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Capture, Learning, and Synthesis of 3D Speaking Styles
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01034 · Cited by 365
AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
Qian, K., Zhang, Y., Chang, S., Yang, X., & Hasegawa-Johnson, M. (2019) · arXiv · doi:10.48550/arxiv.1905.05879 · Cited by 336
Learning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00361 · Cited by 335
A deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303
Realistic Speech-Driven Facial Animation with GANs
Vougioukas, K., Petridis, S., & Pantic, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01251-8 · Cited by 298
Lip Movements Generation at a Glance
Chen, L., Li, Z., Maddox, R. K., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_32 · Cited by 258
Differential Representations for Mesh Processing
Sorkine, O. (2006) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2006.00999.x · Cited by 253
Voice Transformation: A survey
Stylianou, Y. (2009) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2009.4960401 · Cited by 250
Visemenet: audio-driven animator-centric speech animation
Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., & Singh, K. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201292 · Cited by 233
Self-citationFoley Music: Learning to Generate Music from Videos
Gan, C., Huang, D. Y., Chen, P., Tenenbaum, J. B., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_44 · Cited by 203
Background Matting: The World Is Your Green Screen
Sengupta, S., Jayaram, V., Curless, B., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00236 · Cited by 194
Hearing Lips and Seeing Voices
Boston, D. (1977) · British Journal of Audiology · doi:10.3109/03005367709078839 · Cited by 190
Music Gesture for Visual Sound Separation
Gan, C., Huang, D., Zhao, H., Tenenbaum, J. B., & Torralba, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01049 · Cited by 190
Averbuch-Elor, H., Cohen-Or, D., Kopf, J., & Cohen, M. F. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130818 · Cited by 187
JALI: an animator-centric viseme model for expressive lip synchronization
Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925984 · Cited by 184
Talking Face Generation by Conditional Recurrent Adversarial Network
Song, Y., Zhu, J., Li, D., Wang, A., & Qi, H. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/129 · Cited by 182
Self-citationPhoto Wake-Up: 3D Character Animation From a Single Photo
Weng, C.-Y., Curless, B., & Kemelmacher-Shlizerman, I. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00606 · Cited by 160
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer graphics forum (Print) · doi:10.1111/cgf.12552 · Cited by 137
Improvising linguistic style: social and affective bases for agent personality
Walker, M. A., Cahn, J. E., & Whittaker, S. (1997) · doi:10.1145/267658.267680 · Cited by 128
Dancing‐to‐Music Character Animation
Shiratori, T., Nakazawa, A., & Ikeuchi, K. (2006) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2006.00964.x · Cited by 126
The Artistʼs Complete Guide to Facial Expression
Faigin, G., & Leber, D. C. (1991) · Plastic & Reconstructive Surgery · doi:10.1097/00006534-199108000-00037 · Cited by 125
The face of art: landmark detection and geometric style in portraits
Yaniv, J., Newman, Y., & Shamir, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322984 · Cited by 115
FiNet: Compatible and Diverse Fashion Image Inpainting
Han, X., Wu, Z., Huang, W., Scott, M., & Davis, L. S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00458 · Cited by 112
Self-citationExample-based synthesis of stylized facial animations
Fišer, J., Jamriška, O., Simons, D., Shechtman, E., Lu, J., Asente, P., Lukáč, M., & Sýkora, D. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073660 · Cited by 91
Self-citationVideo-audio driven real-time facial animation
Liu, Y., Xu, F., Chai, J.-X., Tong, X., Wang, L., & Huo, Q. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818122 · Cited by 63
Son, J. S., Jamaludin, A., & Zisserman, A. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.109 · Cited by 63
Generating Talking Face Landmarks from Speech
Eskimez, Ş. E., Maddox, R. K., Xu, C., & Duan, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93764-9_35 · Cited by 59
End-to-end Learning for 3D Facial Animation from Speech
Pham, H. X., Wang, Y., & Pavlović, V. (2018) · International Conference on Multimodal Interaction · doi:10.1145/3242969.3243017 · Cited by 56
Rigid stabilization of facial expressions
Beeler, T., & Bradley, D. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601182 · Cited by 54
Neural Style-preserving Visual Dubbing
Kim, H., Elgharib, M., Zollhöfer, M., et al. (2019) · MPG.PuRe (Max Planck Society) · Cited by 50
Noise-Resilient Training Method for Face Landmark Generation From Speech
Eskimez, Ş. E., Maddox, R. K., Xu, C., & Duan, Z. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2947741 · Cited by 35
Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise LossPreprint
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.03820 · Cited by 32
Joint Learning of Facial Expression and Head Pose from Speech
Greenwood, D., Matthews, I. A., & Laycock, S. (2018) · Interspeech · doi:10.21437/interspeech.2018-2587 · Cited by 30
Improvising Linguistic Style: Social and Affective Bases for Agent PersonalityPreprint
Walker, M. A., Cahn, J. E., & Whittaker, S. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9702015 · Cited by 26
VisemeNet: Audio-Driven Animator-Centric Speech AnimationPreprint
Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., & Singh, K. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09488 · Cited by 24
Self-citation
Cited by
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00416 · Cited by 370
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00366 · Cited by 355
Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., & Wang, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00836 · Cited by 336
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., & Cao, X. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530745 · Cited by 155
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., & Pantic, M. (2023) · arXiv · doi:10.48550/arxiv.2301.03396 · Cited by 140
Metadata
- Authors
- Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D.
- Venue
- ACM Transactions on Graphics
- Published
- 2020-11-27
- Volume
- 39
- Issue
- 6
- Pages
- 1-15
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W3019952993
- Cited by
- 343
- References
- 66
- FWCI
- 15.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.