The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Open accessAbstract
Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. Prior works typically focus on learning phoneme-level features of short audio windows with limited context, occasionally resulting in inaccurate lip movements. To tackle this limitation, we propose a Transformer-based autoregressive model, Face-Former, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes. To cope with the data scarcity issue, we integrate the self-supervised pre-trained speech representations. Also, we devise two biased attention mechanisms well suited to this specific task, including the biased cross-modal multi-head (MH) attention and the biased causal MH self-attention with a periodic positional encoding strategy. The former effectively aligns the audio-motion modalities, whereas the latter offers abilities to generalize to longer audio sequences. Extensive experiments and a perceptual user study show that our approach outperforms the existing state-of-the-arts. The code and the video are available at: https://evelynfan.github.io/audio2face/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., & Hu, S.-M. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0229-5 · Cited by 2,100
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,074
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
(2020) · International Conference on Machine Learning · Cited by 1,012
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
End-to-End Human Pose and Mesh Reconstruction with Transformers
Lin, K., Wang, L., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00199 · Cited by 740
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 493
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 1×
“Following ViT, some follow-up works [9, 14, 56] have been introduced to boost performance for image recognition problems.”
BackgroundHierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Capture, Learning, and Synthesis of 3D Speaking Styles
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01034 · Cited by 365
A deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282
Text-based Editing of Talking-head Video
Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323028 · Cited by 270
A Spatio-temporal Transformer for 3D Human Motion Prediction
Aksan, E., Kaufmann, M., Cao, P., & Hilliges, O. (2021) · 2021 International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00066 · Cited by 262
Lip Movements Generation at a Glance
Chen, L., Li, Z., Maddox, R. K., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_32 · Cited by 258
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Richard, A., Zollhofer, M., Wen, Y., de la Torre, F., & Sheikh, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00121 · Cited by 225
Audio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01386 · Cited by 212
JALI: an animator-centric viseme model for expressive lip synchronization
Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925984 · Cited by 184
Real-time facial animation with image-based dynamic avatars
Cao, C., Wu, H., Weng, Y., Shao, T., & Zhou, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925873 · Cited by 172
Expressive speech-driven facial animation
Cao, Y., Tien, W. C., Faloutsos, P., & Pighin, F. (2005) · ACM Transactions on Graphics · doi:10.1145/1095878.1095881 · Cited by 162
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161
A 3-D Audio-Visual Corpus of Affective Communication
Fanelli, G., Gall, J., Romsdorfer, H., Weise, T., & Van Gool, L. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2052239 · Cited by 161
Talking-Head Generation with Rhythmic Head Motion
Chen, L., Cui, G., Liu, C., Li, Z., Kou, Z., Xu, Y., & Xu, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_3 · Cited by 158
Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents
Bhattacharya, U., Rewkowski, N., Banerjee, A., Guhan, P., Bera, A., & Manocha, D. (2021) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr50410.2021.00037 · Cited by 150
TransGAN: Two Transformers Can Make One Strong GANPreprint
Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · Cited by 147
Learning Speech-driven 3D Conversational Gestures from Video
Habibie, I., Xu, W., Mehta, D., Liu, L., Seidel, H., Pons‐Moll, G., Elgharib, M. A., & Theobalt, C. (2021) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3472306.3478335 · Cited by 145
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178899 · Cited by 132
Dynamic Units of Visual Speech
Taylor, S. L., Mahler, M., Theobald, B.-J., & Matthews, I. (2012) · doi:10.2312/sca/sca12/275-284 · Cited by 126
Speech-Driven Facial Animation Using Cascaded GANs for Learning of Motion and Texture
Das, D., Biswas, S., Sinha, S., & Bhowmick, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_25 · Cited by 105
Speech-Driven 3D Facial Animation with Implicit Emotional Awareness: A Deep Learning Approach
Pham, H., Cheung, S., & Pavlović, V. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.287 · Cited by 102
Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00278 · Cited by 101
Learning to Generate Diverse Dance Motions with TransformerPreprint
Li, J., Yin, Y., Chu, H., Zhou, Y., Wang, T., Fidler, S., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08171 · Cited by 75
Video-audio driven real-time facial animation
Liu, Y., Xu, F., Chai, J., Tong, X., Wang, L., & Huo, Q. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818122 · Cited by 56
End-to-end Learning for 3D Facial Animation from Speech
Pham, H., Wang, Y., & Pavlović, V. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243017 · Cited by 51
Animating Face using Disentangled Audio Representations
Mittal, G., & Wang, B. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093527 · Cited by 51
Learn to Dance with AIST++: Music Conditioned 3D Dance GenerationPreprint
Li, R., Yang, S., Ross, D. A., & Kanazawa, A. (2021) · arXiv (Cornell University) · Cited by 45
Modality Dropout for Improved Performance-driven Talking Faces
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418840 · Cited by 42
Geometry-Guided Dense Perspective Network for Speech-Driven Facial Animation
Liu, J., Hui, B., Li, K., Liu, Y., Lai, Y., Zhang, Y., Liu, Y., & Yang, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3107669 · Cited by 31
Speech-driven facial animation with spectral gathering and temporal attention
Chai, Y., Weng, Y., Wang, L., & Zhou, K. (2021) · Frontiers of Computer Science · doi:10.1007/s11704-020-0133-7 · Cited by 17
Moustafa, S., Kirschenmann, W., Dupros, F., & Aochi, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-96983-1_54 · Cited by 15
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
TEMOS: Generating Diverse Human Motions from Textual Descriptions
Petrovich, M., Black, M. J., & Varol, G. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20047-2_28 · Cited by 353
Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., & Wang, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00836 · Cited by 336
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116
Tian, L., Wang, Q., Zhang, B., & Bo, L. (2024) · arXiv · doi:10.48550/arxiv.2402.17485 · Cited by 92
Metadata
- Authors
- Fan, Y., Lin, Z., Saito, J., Wang, W., & Komura, T.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 18749-18758
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4200630629
- Cited by
- 263
- References
- 53
- FWCI
- 86.93
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.