The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
A deep learning approach for generalized speech animation
Open accessAbstract
We introduce a simple and effective deep learning approach to automatically generate natural looking speech animation that synchronizes to input speech. Our approach uses a sliding window predictor that learns arbitrary nonlinear mappings from phoneme label input sequences to mouth movements in a way that accurately captures natural motion and visual coarticulation effects. Our deep learning approach enjoys several attractive properties: it runs in real-time, requires minimal parameter tuning, generalizes well to novel input speech sequences, is easily edited to create stylized and emotional speech, and is compatible with existing animation retargeting approaches. One important focus of our work is to develop an effective approach for speech animation that can be easily integrated into existing production pipelines. We provide a detailed description of our end-to-end approach, including machine learning design decisions. Generalized speech animation results are demonstrated over a wide range of animation clips on a variety of characters and voices, including singing and foreign language input. Our approach can also generate on-demand speech animation in real-time from user speech input.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
(2014) · Cited by 33,419
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,508
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
Active Appearance Models Revisited
Matthews, I., & Baker, S. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029666.37597.d3 · Cited by 1,535
Self-citationYoung, S. J., Jansen, J., Odell, J. J., Ollason, D., & Woodland, P. (1995) · Cited by 1,427
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015736 · Cited by 1,052
Theano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005 · cited 3×
“1We used Keras (http://keras.io/) with Theano [Bastien et al. 2012] ACM Transactions on Graphics, Vol. 36, No. 4, Article 93.”
MethodProceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003
Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · Cited by 680
Video Rewrite: driving visual speech with audio
Bregler, C., Covell, M., & Slaney, M. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258880 · Cited by 669
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 594
Spacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015759 · Cited by 493
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Brand, M. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311537 · Cited by 411
3D shape regression for real-time facial animation
Cao, C., Weng, Y., Lin, S., & Zhou, K. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462012 · Cited by 348
The HMM-based speech synthesis system (HTS) version 2.0.
Zen, H., Nose, T., Yamagishi, J., Sako, S., Masuko, T., Black, A. W., & Tokuda, K. (2007) · SSW · Cited by 343
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566594 · Cited by 301
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., & Gross, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964970 · Cited by 238
Real-time high-fidelity facial performance capture
Cao, C., Bradley, D., Zhou, K., & Beeler, T. (2015) · ACM Transactions on Graphics · doi:10.1145/2766943 · Cited by 217
JALI: an animator-centric viseme model for expressive lip synchronization
Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925984 · Cited by 184
Spacetime faces: high resolution capture for modeling and animation
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1186562.1015759 · Cited by 164
Expressive speech-driven facial animation
Cao, Y., Tien, W. C., Faloutsos, P., & Pighin, F. (2005) · ACM Transactions on Graphics · doi:10.1145/1095878.1095881 · Cited by 162
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566594 · Cited by 142
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178899 · Cited by 132
Dynamic Units of Visual Speech
Taylor, S. L., Mahler, M., Theobald, B.-J., & Matthews, I. (2012) · doi:10.2312/sca/sca12/275-284 · Cited by 126
Self-citationPhoto-realistic talking-heads from image samples
Cosatto, E., & Graf, H. P. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865480 · Cited by 125
Models and Techniques in Computer Animation
Magnenat‐Thalmann, N., & Thalmann, D. (1993) · Springer eBooks · doi:10.1007/978-4-431-66911-1 · Cited by 123
Driving High-Resolution Facial Scans with Video Performance Capture
Fyffe, G., Jones, A., Alexander, O., Ichikari, R., & Debevec, P. E. (2014) · ACM Transactions on Graphics · doi:10.1145/2638549 · Cited by 94
Expressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.434 · Cited by 86
Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
Huang, H., Chai, J., Tong, X., & Wu, H. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964969 · Cited by 85
A Decision Tree Framework for Spatiotemporal Sequence Prediction
Kim, T., Yue, Y., Taylor, S. L., & Matthews, I. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783356 · Cited by 84
Self-citationA Practical and Configurable Lip Sync Method for Games
Xu, Y., Feng, A., Marsella, S., & Shapiro, A. (2013) · doi:10.1145/2522628.2522904 · Cited by 70
A coupled HMM approach to video-realistic speech animation
Xie, L., & Liu, Z. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.12.001 · Cited by 70
Audio/visual mapping with cross-modal hidden Markov models
Fu, S., Gutiérrez‐Osuna, R., Esposito, A. A., Kakumanu, P., & Garcia, O. N. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.843341 · Cited by 60
Accurate visible speech synthesis based on concatenating variable length motion capture data
Ma, J., Cole, R. A., Pellom, B. L., Ward, W., & Wise, B. W. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.18 · Cited by 50
High quality lip-sync animation for 3D photo-realistic talking head
Wang, L., Han, W., & Soong, F. K. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288925 · Cited by 31
Mattheyses, W., Latacz, L., & Verhelst, W. (2013) · Speech Communication · doi:10.1016/j.specom.2013.02.005 · Cited by 28
Real-time facial animation on mobile devices
Weng, Y., Cao, C., Hou, Q., & Zhou, K. (2013) · Graphical Models · doi:10.1016/j.gmod.2013.10.002 · Cited by 24
Deena, S., Hou, S., & Galata, A. (2010) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1891903.1891942 · Cited by 20
Relating Objective and Subjective Performance Measures for AAM-Based Visual Speech Synthesis
Theobald, B.-J., & Matthews, I. A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2202651 · Cited by 14
Self-citation
Cited by
3D Morphable Face Models—Past, Present, and Future
Egger, B., Smith, W. A. P., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3395208 · Cited by 448
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Capture, Learning, and Synthesis of 3D Speaking Styles
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01034 · Cited by 365
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00366 · Cited by 355
Metadata
- Authors
- Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I.
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-11
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2737658251
- Cited by
- 303
- References
- 50
- FWCI
- 19.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.