The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Abstract
We present a machine learning technique for driving 3D facial animation by audio input in real time and with low latency. Our deep neural network learns a mapping from input waveforms to the 3D vertex coordinates of a face model, and simultaneously discovers a compact, latent code that disambiguates the variations in facial expression that cannot be explained by the audio alone. During inference, the latent code can be used as an intuitive control for the emotional state of the face puppet. We train our network with 3--5 minutes of high-quality animation data obtained using traditional, vision-based performance capture methods. Even though our primary goal is to model the speaking style of a single actor, our model yields reasonable results even when driven with audio from other speakers with different gender, accent, or language, as we demonstrate with a user study. The results are applicable to in-game dialogue, low-cost localization, virtual reality avatars, and telepresence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Static Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Bukimi no tani [the uncanny valley]
Mori, M. (1970) · Energy · Cited by 1,485
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
cuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943
Separating Style and Content with Bilinear Models
Tenenbaum, J. B., & Freeman, W. T. (2000) · Neural Computation · doi:10.1162/089976600300015349 · Cited by 840
Automatic speech recognition and speech variability: A review
Benzeghiba, M., De Mori, R., Deroo, O., Dupont, S., Erbes, T., Jouvet, D., Fissore, L., Laface, P., Mertins, A., Ris, C., et al. (2007) · Speech Communication · doi:10.1016/j.specom.2007.02.006 · Cited by 529
Modeling Coarticulation in Synthetic Visual Speech
Cohen, M. M., & Massaro, D. W. (1993) · Models and Techniques in Computer Animation · doi:10.1007/978-4-431-66911-1_13 · Cited by 445
Brand, M. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311537 · Cited by 411
Multilinear subspace analysis of image ensembles
Vasilescu, M. A. O., & Terzopoulos, D. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211457 · Cited by 375
Confusions Among Visually Perceived Consonants
Fisher, C. G. (1968) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1104.796 · Cited by 335
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566594 · Cited by 301
Practice and Theory of Blendshape Facial Models
Lewis, J. P., Anjyo, K., Rhee, T. J., Zhang, M., Pighin, F. P., & Deng, Z. (2014) · Eurographics · doi:10.2312/egst.20141042 · Cited by 204
JALI: an animator-centric viseme model for expressive lip synchronization
Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925984 · Cited by 184
Separating style and content on a nonlinear manifold
Elgammal, A., & Lee, C.-S. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315070 · Cited by 182
Virtual character performance from speech
Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A. (2013) · Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/2485895.2485900 · Cited by 168 · cited 1×
“Several authors have tried alleviating the effect by deducing additional motion, such as head movements, eye saccades or blinks from audio (e.g., [Deng et al. 2004; Marsella et al. 2013]).”
BackgroundExpressive speech-driven facial animation
Cao, Y., Tien, W. C., Faloutsos, P., & Pighin, F. (2005) · ACM Transactions on Graphics · doi:10.1145/1095878.1095881 · Cited by 162
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566594 · Cited by 142
Dynamic Units of Visual Speech
Taylor, S. L., Mahler, M., Theobald, B.-J., & Matthews, I. (2012) · doi:10.2312/sca/sca12/275-284 · Cited by 126
Automated lip-synch and speech synthesis for character animation
Lewis, J. P., & Parke, F. I. (1986) · doi:10.1145/29933.30874 · Cited by 99
Real-time speech-driven face animation with expressions using neural networks
Hong, P., Wen, Z., & Huang, T. S. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021892 · Cited by 99
Schwartz, J., & Savariaux, C. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003743 · Cited by 98
Automated lip‐sync: Background and techniques
Lewis, J. P. (1991) · The Journal of Visualization and Computer Animation · doi:10.1002/vis.4340020404 · Cited by 90
Facial expression space learning
Chuang, E. S., Deshpande, F., & Bregler, C. (2003) · Pacific Conference on Computer Graphics and Applications · doi:10.1109/pccga.2002.1167840 · Cited by 90
Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces
Deng, Z., Neumann, U., Lewis, J. P., Kim, T.-Y., Bulut, M., & Narayanan, S. B. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.90 · Cited by 87
Audiovisual speech synthesis: An overview of the state-of-the-art
Mattheyses, W., & Verhelst, W. (2014) · Speech Communication · doi:10.1016/j.specom.2014.11.001 · Cited by 87
Expressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.434 · Cited by 86
A deep bidirectional LSTM approach for video-realistic talking head
Fan, B., Xie, L., Yang, S., Wang, L., & Soong, F. K. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2944-3 · Cited by 66
Massaro, D. W., Cohen, M. M., Tabain, M., Beskow, J., & Clark, R. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511843891.014 · Cited by 51
Audio-based head motion synthesis for Avatar-based telepresence systems
Deng, Z., Narayanan, S., Busso, C., & Neumann, U. (2004) · doi:10.1145/1026776.1026784 · Cited by 33
Joint Audiovisual Hidden Semi-Markov Model-Based Speech Synthesis
Schabus, D., Pucher, M., & Hofer, G. (2013) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2013.2281036 · Cited by 33
Realistic facial expression synthesis for an image-based talking head
Liu, K., & Östermann, J. (2011) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2011.6011835 · Cited by 32
Audio-to-Visual Speech Conversion Using Deep Neural Networks
Taylor, S. L., Kato, A., Matthews, I. A., & Milner, B. P. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-483 · Cited by 30
Head and facial gestures synthesis using PAD model for an expressive talking avatar
Jia, J., Wu, Z., Zhang, S., Meng, H. M. L., & Cai, L. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-013-1604-8 · Cited by 29
HMM trajectory-guided sample selection for photo-realistic talking head
Wang, L., & Soong, F. K. (2014) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2118-8 · Cited by 23
Speech-Driven Facial Animation Using a Shared Gaussian Process Latent Variable Model
Deena, S., & Galata, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10331-5_9 · Cited by 17
How well can People and Computers Recognize Emotions in Speech
Petrushin, V. A. (1998) · Cited by 12
Visual Speech Synthesis Using a Variable-Order Switching Shared Gaussian Process Dynamical Model
Deena, S., Hou, S., & Galata, A. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2279659 · Cited by 12
Emphatic Visual Speech Synthesis
Melenchón, J., Marroquin, E. M., la Torre, F. D., & Montero, J. A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2010213 · Cited by 12
Cited by
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Voxceleb: Large-scale speaker verification in the wild
Nagrani, A., Chung, J. S., Xie, W., & Zisserman, A. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2019.101027 · Cited by 660
3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437
X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose Codes
Wiles, O., Koepke, A. S., & Zisserman, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_41 · Cited by 435
Metadata
- Authors
- Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J.
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2739192055
- Cited by
- 434
- References
- 43
- FWCI
- 18.60
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.