Modality Dropout for Improved Performance-driven Talking Faces
Open accessAbstract
We describe our novel deep learning approach for driving animated faces using both acoustic and visual information. In particular, speech-related facial movements are generated using audiovisual information, and non-verbal facial movements are generated using only visual information. To ensure that our model exploits both modalities during training, batches are generated that contain audio-only, video-only, and audiovisual input features. The probability of dropping a modality allows control over the degree to which the model exploits audio and visual information during training. Our trained model runs in real-time on resource limited hardware (e.g. a smart phone), it is user agnostic, and it is not dependent on a potentially error-prone transcription of the speech. We use subjective testing to demonstrate: 1) the improvement of audiovisual-driven animation over the equivalent video-only approach, and 2) the improvement in the animation of speech-related facial movements after introducing modality dropout. Without modality dropout, viewers prefer audiovisual-driven animation in 51% of the test sequences compared with only 18% for video-driven. After introducing dropout viewer preference for audiovisual-driven animation increases to 74%, but decreases to 8% for video-only.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current methods for robust multimodal learning require retraining the whole model with a specialized training strategy or utilize an extra module/sub-network to guide the multimodal model.
“The current methods for robust MML, as discussed in Section 2, require retraining the whole model with specialized training strategy [20, 18] or utilize extra module/sub-network to guide the multi-modal model [22, 21].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [18]
The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using a specialized training strategy.
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057 · see citing works
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · see citing works
Video Rewrite: driving visual speech with audio
Bregler, C., Covell, M., & Slaney, M. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258880 · Cited by 669 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
Canonical Correlation Analysis
Thompson, B. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa068 · Cited by 470 · see citing works
Realtime performance-based facial animation
Weise, T., Bouaziz, S., Li, H., & Pauly, M. V. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964972 · Cited by 461 · see citing works
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 435 · see citing works
Performance-driven facial animation
Williams, L. R. (2006) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1185657.1185856 · Cited by 423 · see citing works
ModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420 · see citing works
Brand, M. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311537 · Cited by 412 · see citing works
Displaced dynamic expression regression for real-time facial tracking and animation
Cao, C., Hou, Q., & Zhou, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601204 · Cited by 378 · see citing works
Capture, Learning, and Synthesis of 3D Speaking Styles
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01034 · Cited by 371 · see citing works
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/566570.566594 · Cited by 304 · see citing works
A deep learning approach for generalized speech animation
Taylor, S. L., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J. K., & Matthews, I. A. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303 · see citing works
Gated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 302 · see citing works
Online modeling for realtime facial animation
Bouaziz, S., Wang, Y., & Pauly, M. V. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461976 · Cited by 288 · see citing works
Realtime facial animation with on-the-fly correctives
Li, H., Yu, J., Ye, Y., & Bregler, C. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2462019 · Cited by 282 · see citing works
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2018) · Communications of the ACM · doi:10.1145/3292039 · Cited by 244 · see citing works
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P. A., Gotsman, C., Sumner, R. W., & Groß, M. H. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964970 · Cited by 239 · see citing works
High-quality passive facial performance capture using anchor frames
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P. A., Gotsman, C., Sumner, R. W., & Groß, M. H. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964970 · Cited by 224 · see citing works
Real-time high-fidelity facial performance capture
Cao, C., Bradley, D., Zhou, K., & Beeler, T. (2015) · ACM Transactions on Graphics · doi:10.1145/2766943 · Cited by 217 · see citing works
Li, H., Weise, T., & Pauly, M. V. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778769 · Cited by 196 · see citing works
Talking Face Generation by Conditional Recurrent Adversarial Network
Song, Y., Zhu, J., Li, D., Wang, A. Y.-D., & Qi, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/129 · Cited by 192 · see citing works
Alexander, O., Mike Rogers, A., Lambeth, W., Chiang, M. J., & Debevec, P. E. (2009) · ACM SIGGRAPH Courses · doi:10.1145/1667239.1667251 · Cited by 131 · see citing works
Trainable videorealistic speech animation
Ezzat, T., Geiger, G., & Poggio, T. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566594 · Cited by 130 · see citing works
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 127 · see citing works
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178899 · Cited by 126 · see citing works
Audio-visual based emotion recognition-a new approach
Song, M., Bu, J., Chen, C., & Li, N. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315276 · Cited by 126 · see citing works
Photo-realistic talking-heads from image samples
Cosatto, E., & Graf, H. P. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865480 · Cited by 125 · see citing works
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682524 · Cited by 108 · see citing works
Production-level facial performance capture using deep convolutional neural networks
Laine, S., Karras, T., Aila, T., Herva, A., Saito, S., Yu, R., Li, H., & Lehtinen, J. (2017) · ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) · doi:10.1145/3099564.3099581 · Cited by 96 · see citing works
End-to-End Speech-Driven Facial Animation with Temporal GANsPreprint
Vougioukas, K., Petridis, S., & Pantić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09313 · Cited by 92 · see citing works
An anatomically-constrained local deformation model for monocular face capture
Wu, C., Bradley, D., Groß, M. H., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925882 · Cited by 88 · see citing works
Expressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.434 · Cited by 85 · see citing works
Speech-driven facial animation with realistic dynamics
Gutiérrez‐Osuna, R., Kakumanu, P., Esposito, A. A., et al. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2004.840611 · Cited by 75 · see citing works
Realistic Mouth-Synching for Speech-Driven Talking Face Using Articulatory Modelling
Xie, L., & Liu, Z.-Q. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.888009 · Cited by 75 · see citing works
A coupled HMM approach to video-realistic speech animation
Xie, L., & Liu, Z. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.12.001 · Cited by 70 · see citing works
A Decision Tree Framework for Spatiotemporal Sequence Prediction
Kim, T., Yue, Y., Taylor, S. L., & Matthews, I. A. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783356 · Cited by 69 · see citing works
Learning Dynamic Stream Weights For Coupled-HMM-based Audio-visual Speech Recognition
Abdelaziz, A. H., Zeiler, S., & Kolossa, D. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2409785 · Cited by 65 · see citing works
A Groupwise Multilinear Correspondence Optimization for 3D Faces
Bolkart, T., & Wuhrer, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.411 · Cited by 62 · see citing works
Audio/visual mapping with cross-modal hidden Markov models
Fu, S., Gutiérrez‐Osuna, R., Esposito, A. A., Kakumanu, P., & Garcia, O. N. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.843341 · Cited by 60 · see citing works
Video-audio driven real-time facial animation
Liu, Y., Xu, F., Chai, J., Tong, X., Wang, L., & Huo, Q. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818122 · Cited by 55 · see citing works
Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks
Sadoughi, N., & Busso, C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2916031 · Cited by 50 · see citing works
End-to-end Learning for 3D Facial Animation from Speech
Pham, H., Wang, Y., & Pavlović, V. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243017 · Cited by 49 · see citing works
Shon, S., Oh, T.-H., & Glass, J. R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683477 · Cited by 46 · see citing works
Hidden Markov Model Inversion for Audio-to-Visual Conversion in an MPEG-4 Facial Animation System
Choi, K.-H., Luo, Y., & Hwang, J. (2001) · The Journal of VLSI Signal Processing Systems for Signal Image and Video Technology · doi:10.1023/a:1011171430700 · Cited by 45 · see citing works
A Fully End-to-End Cascaded CNN for Facial Landmark Detection
He, Z., Kan, M., Zhang, J., et al. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.33 · Cited by 41 · see citing works
Comparing Fusion Models for DNN-Based Audiovisual Continuous Speech Recognition
Abdelaziz, A. H. (2018) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2783545 · Cited by 41 · see citing works
Audio-to-Visual Speech Conversion Using Deep Neural Networks
Taylor, S. L., Kato, A., Matthews, I. A., & Milner, B. P. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-483 · Cited by 30 · see citing works
Mattheyses, W., Latacz, L., & Verhelst, W. (2013) · Speech Communication · doi:10.1016/j.specom.2013.02.005 · Cited by 28 · see citing works
Speech-Driven Facial Reenactment Using Conditional Generative Adversarial NetworksPreprint
Jalalifar, S. A., Hasani, H., & Aghajan, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.07461 · Cited by 24 · see citing works
Talking heads synthesis from audio with deep neural networks
Shimba, T., Sakurai, R., Yamazoe, H., & Lee, J. (2015) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2015.7404961 · Cited by 22 · see citing works
Talking Face Generation by Conditional Recurrent Adversarial NetworkPreprint
Song, Y., Zhu, J., Li, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04786 · Cited by 20 · see citing works
3-D Face Point Trajectory Synthesis Using An Automatically Derived Visual Phoneme Similarity Matrix.
Arslan, L. M., & Talkin, D. T. (1998) · Cited by 16 · see citing works
Expressive visual text to speech and expression adaptation using deep neural networks
Parker, J., Maia, R., Stylianou, Y., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953092 · Cited by 14 · see citing works
End-to-end Learning for 3D Facial Animation from Raw Waveforms of SpeechPreprint
Pham, H., Wang, Y., & Pavlović, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.00920 · Cited by 14 · see citing works
A real-time text to audio-visual speech synthesis system
Wang, L., Qian, X., Ma, L., et al. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-596 · Cited by 11 · see citing works
Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models
Abdelaziz, A. H., Theobald, B.-J., Binder, J., Fanelli, G., Dixon, P. R., Apostoloff, N., Weise, T., & Kajareker, S. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353745 · Cited by 8 · see citing works
Self-citationShon, S., Oh, T.-H., & Glass, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10813
Flagged first, then by citations
Research works printed but not linked: 4
- [16]16. J. Chung, A. Senior, O. Vinyals, and A. Zisserman. Lip reading sentences in the wild. In Conference on Computer Vision and Pattern Recognition, pages 3444-3453, 2016.
- [48]48. S. Taylor, M. Mahler, B. Theobald, and I. Matthews. Dynamic units of visual speech. In Symposium on Computer Animation, pages 275-284, 2012.
- [49]49. J. Thies, M. Elgharib, A. Tewari, C. Theobalt, and M. Niessner. Neural voice puppetry: Audio- driven facial reenactment. arXiv 2019, 2019.
- [50]50. J. Thies, M. Zollhoefer, M. Stamminger, C. Theobalt, and M. Niessner. Face2Face: Real-time face capture and reenactment of RGB videos. Communications of the ACM, 62(1):96-104, 2018.
Cited by
Sorted by citations
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 61 · see citing works
“For instance, one approach is to adopt robust training strategies such as modality dropout during training [42, 25], partial or full modality masking [2, 48], and knowledge distillation [52, 40].”
BackgroundMethodInfluentialMissing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
Maheshwari, H., Liu, Y.-C., & Kira, Z. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00106 · Cited by 37 · see citing works
- Authors
- Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S.
- Affiliations
- Apple, Cupertino, CA, USA
- Apple (United States) (US)
- Venue
- Proceedings of the 2020 International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2020-10-21
- Pages
- 378-386
- OpenAlex
- W3029296930
- Cited by
- 42
- References
- 62
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 16 · 2023: 10 · 2022: 4 · 2021: 4
- FWCI
- 2.80
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- modality dropout, user preferences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
