The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Capture, Learning, and Synthesis of 3D Speaking Styles
Open accessAbstract
Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Hermansky, H., & Morgan, N. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.326616 · Cited by 1,808
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01123 · Cited by 1,781
Self-citationBulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464
Learning a model of facial shape and expression from 4D scans
Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130813 · Cited by 1,223
Self-citationA 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023
Bosphorus Database for 3D Face Analysis
Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89991-4_6 · Cited by 812
BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database
Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.002 · Cited by 745
Video Rewrite: driving visual speech with audio
Bregler, C., Covell, M., & Slaney, M. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258880 · Cited by 669
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620
Demo of Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016) · ACM SIGGRAPH 2016 Emerging Technologies · doi:10.1145/2929464.2929475 · Cited by 589
A high-resolution 3D dynamic facial expression database
Yin, L., Chen, X., Sun, Y., Worm, T., & Reale, M. (2008) · 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/afgr.2008.4813324 · Cited by 488
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964972 · Cited by 460
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis
Zhang, Z., Girard, J. M., Wu, Y., Zhang, X., Liu, P., Ciftci, U., Canavan, S., Reale, M., Horowitz, A., Yang, H., et al. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.374 · Cited by 424
Brand, M. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311537 · Cited by 411
Displaced dynamic expression regression for real-time facial tracking and animation
Cao, C., Hou, Q., & Zhou, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601204 · Cited by 356
A deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566594 · Cited by 301
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 273
Lip Movements Generation at a Glance
Chen, L., Li, Z., Maddox, R. K., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_32 · Cited by 258
Visemenet: audio-driven animator-centric speech animation
Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., & Singh, K. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201292 · Cited by 233
Real-time high-fidelity facial performance capture
Cao, C., Bradley, D., Zhou, K., & Beeler, T. (2015) · ACM Transactions on Graphics · doi:10.1145/2766943 · Cited by 217
Deformation transfer for triangle meshes
Sumner, R. W., & Popović, J. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015736 · Cited by 203
JALI: an animator-centric viseme model for expressive lip synchronization
Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925984 · Cited by 184
Rigid Head Motion in Expressive Speech Animation: Analysis and Synthesis
Busso, C., Deng, Z., Grimm, M., Neumann, U., & Narayanan, S. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2006.885910 · Cited by 180
Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., & Pfister, H. (2011) · Proceedings of the 2011 SIGGRAPH Asia Conference · doi:10.1145/2024156.2024164 · Cited by 166
Expressive speech-driven facial animation
Cao, Y., Tien, W. C., Faloutsos, P., & Pighin, F. (2005) · ACM Transactions on Graphics · doi:10.1145/1095878.1095881 · Cited by 162
Cosker, D. P., Krumhuber, E. G., & Hilton, A. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126510 · Cited by 162
A 3-D Audio-Visual Corpus of Affective Communication
Fanelli, G., Gall, J., Romsdorfer, H., Weise, T., & Van Gool, L. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2052239 · Cited by 161
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 133
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178899 · Cited by 132
4DFAB: A Large Scale 4D Database for Facial Expression Analysis and Biometric Applications
Cheng, S., Kotsia, I., Pantić, M., & Zafeiriou, S. P. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00537 · Cited by 104
Production-level facial performance capture using deep convolutional neural networks
Laine, S., Karras, T., Aila, T., Herva, A., Saito, S., Yu, R., Li, H., & Lehtinen, J. (2017) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/3099564.3099581 · Cited by 102
Real-time speech-driven face animation with expressions using neural networks
Hong, P., Wen, Z., & Huang, T. S. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021892 · Cited by 100
Speech-Driven 3D Facial Animation with Implicit Emotional Awareness: A Deep Learning Approach
Pham, H., Cheung, S., & Pavlović, V. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.287 · Cited by 89
Expressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.434 · Cited by 86
An anatomically-constrained local deformation model for monocular face capture
Wu, C., Bradley, D., Groß, M. H., & Beeler, T. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925882 · Cited by 86
Realistic Mouth-Synching for Speech-Driven Talking Face Using Articulatory Modelling
Xie, L., & Liu, Z.-Q. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.888009 · Cited by 75
Face transfer with multilinear models
Vlasic, D., Brand, M., Pfister, H., & Popović, J. (2005) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1186822.1073209 · Cited by 75
Automatic 3D Facial Expression Analysis in Videos
Chang, Y., Vieira, M., Turk, M., & Velho, L. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564386_23 · Cited by 74
A deep bidirectional LSTM approach for video-realistic talking head
Fan, B., Xie, L., Yang, S., Wang, L., & Soong, F. K. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2944-3 · Cited by 61
A Groupwise Multilinear Correspondence Optimization for 3D Faces
Bolkart, T., & Wuhrer, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.411 · Cited by 61
Self-citationHead motion synthesis from speech using deep neural networks
Ding, C., Xie, L., & Zhu, P. (2014) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2156-2 · Cited by 57
Video-audio driven real-time facial animation
Liu, Y., Xu, F., Chai, J., Tong, X., Wang, L., & Huo, Q. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818122 · Cited by 56
SynFace—Speech-Driven Facial Animation for Virtual Speech-Reading Support
Salvi, G., Beskow, J., Moubayed, S. A., et al. (2009) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1155/2009/191940 · Cited by 39
Talking heads synthesis from audio with deep neural networks
Shimba, T., Sakurai, R., Yamazoe, H., & Lee, J. (2015) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2015.7404961 · Cited by 22
A 3D Dynamic Database for Unconstrained Face Recognition
Alashkar, T., Amor, B. B., Daoudi, M., & Berretti, S. (2014) · International Conference on 3D Body Scanning Technologies (3DBST) · doi:10.15221/14.357 · Cited by 11
Cited by
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384
AI-generated characters for supporting personalized learning and well-being
Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00417-9 · Cited by 358
DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
Shen, S., Zhao, W., Meng, Z., Li, W., Zhu, Z., Zhou, J., & Lu, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00197 · Cited by 133
Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00278 · Cited by 101
Prediction of weaning from mechanical ventilation using Convolutional Neural Networks
Jia, Y., Kaul, C., Lawton, T., Murray-Smith, R., & Habli, I. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102087 · Cited by 79
Metadata
- Authors
- Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 10093-10103
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2944661720
- Cited by
- 365
- References
- 50
- FWCI
- 36.74
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
