Deep Learning for Visual Speech Analysis: A Survey
Open accessThe DOI matches its Crossref record; 188 of the 189 references it cites were found in a registry or an index, and 1 are not checked yet; 10 of the 273 sources its claims cite were checked for support.
Abstract
Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.
Verified 2026-10-11 · 1 reference not checked yet
Overview
CoChat Summary
## TL;DR
This survey reviews deep learning methods for visual speech analysis, focusing on the two dual tasks of visual speech recognition (VSR) and visual speech generation (VSG). It summarizes datasets, challenges, evaluation metrics, representative methods, and reported state-of-the-art results, and it highlights open problems and future directions.
## Key findings
- The paper frames VSA around two formally dual problems: VSR and VSG, and argues that VSR and VSG share core technical points such as visual speech representation learning and contextual sequence modeling.
- For VSR, the paper reports strong recent performance: the SOTA on LRW reached 88.5% and on LRW-1000 50.5% with cross-modal mutual learning; on the simple GRID dataset, deep VSR models almost saturate at 1.3% WER.
- For word-level VSR, a baseline C3D-ResNet34 + BiLSTM achieved 83.5% on LRW, and later methods improved to 87.7% on LRW; MS-TCN-based methods are reported at 85.3% on LRW and 41.4% on LRW-1000.
- For VSG, the paper compares representative methods using PSNR, SSIM, and LMD, and reports examples such as 29.36 PSNR / 0.74 SSIM / 1.35 LMD for a one-stage method on VoxCeleb+LRW, and 33.4 PSNR / 0.96 SSIM / 0.60 LMD for Prajwal et al. on LRS2.
## Methods
- Survey design: a comprehensive literature review organized around two main tasks, VSR and VSG, plus datasets, evaluation metrics, and practical challenges.
- Coverage: discusses benchmark datasets in controlled and uncontrolled settings, with examples including AVICAR, GRID, OuluVS2, LRW (> 1,000k word instances), LRS2-BBC (> 144.5k utterances), VoxCeleb1 (> 100k utterances from 1,251 celebrities), LRS3-TED (> 400 hours), and MuAViC (1,200 hours in 9 languages).
- Analysis: synthesizes methods by taxonomy and reports representative performance comparisons from published results rather than conducting new experiments.
## Limitations
- The paper explicitly says that many practical issues remain unresolved, including the need for fewer labels, better benchmark fairness, improved robustness analysis, and better evaluation benchmarks for VSG.
- It states that fair comparison is difficult because VSR/VSG results are affected by extra language models, extra training datasets, audio signals, implementation details, and non-unified test platforms.
- Evident from the text: this is a survey, so it does not present a new experimental dataset, model, or unified benchmark of its own; instead it depends on heterogeneous reported results from prior work.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 50 Not valid · 176 Not checked · 132 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Meanwhile, the emergence of large-scale VSA datasets [28, 29, 30, 31, 32] promoted the further development of deep learning based VSA research.”
What the sources say
- Not checked yetcites [28] source
- Not checked yetcites [29] source
- Supportedcites [30] source
The Introduction (p. 1) presents VoxCeleb2 as a large-scale speaker-recognition dataset addressing a shortage of such data. The paper also reports training a deep CNN speaker-recognition system on it and achieving state-of-the-art verification performance (p. 1; Conclusion, p. 4). This supports its use as an example of a large-scale dataset advancing deep-learning-based speaker recognition. (LLM, conf=0.90)
- Supportedcites [31] source
The Abstract identifies VoxCeleb as a large-scale speaker dataset and reports CNN-based identification and verification results on it. Related Works says a large real-world dataset is essential to the shift toward deep CNNs for speaker identification, supporting the cited connection between such datasets and deep-learning research. (LLM, conf=0.91)
- Not checked yetcites [32] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Many deep learning driven technologies [39, 105, 106, 107] have been introduced to address the alignment of AV signals in a self-supervised way, i . e ., automatically creating positive and negative A-V sync/out-of-sync pairs.”
What the sources say
- Supportedcites [39] source
Section 3.3.1 describes an audio-video sync discriminator trained with automatically sampled in-sync and out-of-sync pairs: an audio window is taken either from the matching time or a different time. Section 3.3.2 says the paper adapts and trains that discriminator for its own lip-generation method, supporting its use as an example of the technology described. (LLM, conf=0.90)
- Not checked yetcites [105] source
- Not checked yetcites [106] source
- Not checked yetcites [107] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“[11] is the first work introducing transformers to VSR.”
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 3 (page unavailable) describes Transformer-based audio-visual speech-recognition models, supporting the claim that this work uses Transformers for VSR. But the Introduction (page unavailable) says one Transformer variant had already been published in [2]; our copy does not establish that this paper was the first to introduce Transformers to VSR. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
The Introduction and §6.1 (Lips only; page unavailable) report that the Transformer-based model improved sentence-level visual speech recognition from 70.4% to 48.3% word error rate relative to the earlier model. But the Introduction says the visual frontend was also updated and attributes the improvement to both frontend and backend changes; the experiments do not isolate a Transformer-versus-RNN backend effect. (LLM, conf=0.87)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Except for marginbased loss, L1 loss and binary classification loss [48, 149, 150, 151] are also widely used for A-V representations learning.”
What the sources say
- Not checked yetcites [48] source
- Partly supportedcites [149] sourceExperimental — accuracy not yet measured
Our copy of Senocak et al. (CVPR 2018) supports only the binary-classification half of the claim: in Sec. 4 (Eq. 7) the supervised loss L_S is a cross-entropy over the attention map whose ground-truth values α_GT,i are binary, and the paper notes it performs slightly better than the mean-square-error alternative. The L1-loss part is not in our copy — the unsupervised objective is the distance-ratio/triplet loss (Eq. 5, Sec. 4) built on ℓ2 distances, and no ℓ1 loss appears anywhere (searched: L1 loss, distance/embedding objective, loss ablation). The paper also makes no statement that either loss is 'widely used' for audio-visual representation learning; it is a single method paper, so the survey-level framing rests on the other citations in the cluster. (LLM, conf=0.68)
- Not checked yetcites [150] source
- Not checked yetcites [151] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
97 more claims · 1 with their cited sources checked
Create a free account to see all 367 claims and to verify the 263 references still open.
- Probably not a substantial claim. 216
- It may state two separate claims as one. 106
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1.2 Datasets under uncontrolled environments
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
§ 3.1.2 Datasets under uncontrolled environments
“Compared to LRS2-TED, it has a larger scale in terms of duration, vocabulary, and number of speakers.”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Representative CNN architectures, such as VGG [124], ResNet [125], MobileNet [126], DenseNet [127] etc, have been widely used in learning visual representation for VSR.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformer-Based Architectures: Compared to RNN-based architectures, Transformers [136] have significant advantages in long-term dependency and parallel computation.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 5×
“Representative CNN architectures, such as VGG [121], ResNet [122], MobileNet [123], DenseNet [124], ShuffleNet [125] etc, have been widely used in learning visual representation for VSR.”
MethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Since AlexNet [26] was proposed in 2012, researchers have invented a variety of deeper, wider, and Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“[64] proposed the LSTM + UNet architecture, improving the model by replacing the complex video generation pipeline with a pix2pix framework [161].”
MethodMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · cited 1×
“In 2018, Jalalifar et al. [161] proposed LSTM + C-GAN VSG backbone architecture, using the basic conditional generative adversarial network (C-GAN) [162] for generating talking faces given the learned landmarks.”
MethodArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · cited 1×
“Besides, Zakharov et al. [117] used the cosine similarity between embedding vectors of the ArcFace network [118] for measuring identity mismatch.”
MethodConvolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · cited 3×
“Wang et al.integrated the attention mechanism into the model to further improve the BiConvLSTM architecture.”
BackgroundMethodInfluentialHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 1×
“intrinsically bimodal, as shown several decades ago by the famous McGurk effect [2] that human speech perception depends not only on auditory information, but also on visual cues like lip movements.”
BackgroundSpatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · cited 1×
“As we know, most of the existing VSR models are derived from general backbone models used in other fields (e.g., action recognition [137], [138], audio speech recognition [135], etc.), and few are designed explicitly for VSR.”
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 1×
“[149] are the first to train an A-V synchronization model in an end-to-end manner with margin-based [150] pairwise contrastive loss.”
BackgroundA Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084827 · Cited by 5,273 · cited 1×
“Restrictions apply. lighter CNN models [123].”
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · cited 1×
“Visual Transformer-Based Architectures: Inspired by the significant success of transformer architectures in the field of NLP, researchers have recently applied transformers to computer vision (CV) tasks [132].”
MethodFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 1×
“Faceforensics++ [94] is an automated benchmark for facial manipulation detection.”
BackgroundVoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · cited 1×
“Chan et al. [141] proposed an elegant seq2seq methodtotranscribeaudiosignalstocharacters.”
BackgroundVoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · cited 2×
“VoxCeleb1 [30] is a large-scale text-independent audio-visual dataset collected from open-source YouTube media.”
BackgroundFace2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
An audio-visual corpus for speech perception and automatic speech recognition
Cooke, M., Barker, J., Cunningham, S., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,144 · cited 2×
“GRID [80], consisting of high-quality audio and video recordings of 1,000 syntactically identical phrases spoken by 34 talkers, is built for comprehensive audio-visual perceptual analysis and microscopic modeling.”
BackgroundMethodMoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
FaceWarehouse: A 3D Facial Expression Database for Visual Computing
Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.249 · Cited by 1,023 · cited 1×
“Pham et al. [168], [169], [173] proposed utilizing CNN + RNN based backbone architectures to map audio signals to blendshape coefficients [174] of a 3D face.”
MethodA Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897 · cited 4×
“More recently, Prajwal et al. [39] introduced Fr ´e chet Inception Distance (FID) to measure the distance between synthetic and real data distributions, as FID is more consistent with human perception evaluation.”
BackgroundMethodInfluentialRistad, E. S., & Yianilos, P. N. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.682181 · Cited by 859 · cited 1×
“Asforthesentence-leveltask,CharacterErrorRate(CER)and WordErrorRate(WER)[114],alsoknownasaveragecharacter-levelandword-leveleditdistances,arethemostcommonlyused evaluationmetrics.”
(2016) · Cited by 844
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · cited 6×
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
BackgroundMethodInfluentialAccurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00038 · Cited by 763 · cited 1×
“Besides the 3D blendshape model, Kim et al. [175], [176] introduced 3D Morphable Model (3DMM) [177], a more dense 3 d face parametric representation, for video-oriented face2face translation.”
MethodRecent advances in the automatic recognition of audiovisual speech
Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817150 · Cited by 752 · cited 1×
“In the era dominated by traditional VSA methods, shallow representations of visual speech such as visemes [19], mouth geometry descriptors [20], linear transformation features [21], statistical representations [22], and sequence modeling like Gaussian process dynamical models [23], hidden Markov…”
MethodAudio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
Lip reading sentences in the wild
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.367 · Cited by 702 · cited 4×
“Many works [29], [51], [56], [57], [129], [133] have adopted RNN-based (BiLSTM or BiGRU) network architectures as the temporal backend network in VSR.”
BackgroundMethodInfluentialOut of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · cited 3×
“Many deep learning driven technologies [39], [105], [106], [107] have been introduced to address the alignment of A-V signals in a self-supervised way, i.e., automatically creating positive and negative A-V sync/out-of-sync pairs.”
BackgroundMethodInfluentialFew-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687 · cited 1×
“Besides, Zakharov et al. [117] used the cosine similarity between embedding vectors of the ArcFace network [118] for measuring identity mismatch.”
MethodVideo Rewrite: driving visual speech with audio
Bregler, C., Covell, M., & Slaney, M. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258880 · Cited by 669
Audio-visual speech modeling for continuous speech recognition
Dupont, S., & Luettin, J. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865479 · Cited by 598 · cited 2×
“From the feature engineering perspective, traditional feature extraction methods can be categorized into three types: appearance-based, shape-based, and motion-based [12].”
MethodFirst Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585 · cited 1×
“Finally, the synthesized talking video is obtained using a first-order motion model [164].”
MethodExtraction of visual features for lipreading
Matthews, I., Cootes, T. F., Bangham, J. A., Cox, S., & Harvey, R. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982900 · Cited by 537 · cited 1×
“…of visual speech such as visemes [19], mouth geometry descriptors [20], linear transformation features [21], statistical representations [22], and sequence modeling like Gaussian process dynamical models [23], hidden Markov models (HMMs) [24], decision tree models [25] 0162-8828 © 2024…”
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 5×
“LRW [28] is a word-level audio-visual dataset constructed by a multi-stage data automatic collection pipeline.”
BackgroundMethodInfluentialLips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00500 · Cited by 537 · cited 1×
“Inpublicsecurity,VSAcanbe appliedtofaceforgerydetection[14]andlivenessdetection[15].”
Transfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519 · cited 1×
“Typical academia and practical applications of VSA include multimodal speech recognition and enhancement, speaker recognition and verification [3], medical assistance, security, forensic, video compression, entertainment, humancomputer interaction, emotion understanding [4, 5], etc.”
BackgroundDeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 1×
“Different from existing audio-visual datasets, all videos have been manipulated based on Deep-Fakes [108], Face2Face [109], FaceSwap [110], NeuralTex-tures [111] as main methods for facial manipulations.”
MethodKim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 475 · cited 1×
“Besides the 3D blendshape model, Kim et al. [175], [176] introduced 3D Morphable Model (3DMM) [177], a more dense 3 d face parametric representation, for video-oriented face2face translation.”
MethodHierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437 · cited 1×
“In 2019, Chen et al. [66] proposed a novel LSTM + Convolutional-RNN structure, further considering the correlation between adjacent video frames during generation.”
BackgroundAudio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434 · cited 2×
“…security,forensic,videocompression,entertainment,human-computerinteraction,emotionunderstanding[9],[10],etc. Togivesomeapplicationexamples,inspeechrecognitionand enhancement,visualspeechcanbetreatedasacomplementary…”
MethodTalking Face Generation by Adversarially Disentangled Audio-Visual Representation
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416
Facial Landmark Detection: A Literature Survey
Wu, Y., & Ji, Q. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1097-z · Cited by 404 · cited 1×
“1) Landmark Based Methods: Facial landmark points around facial components capture the rigid and non-rigid facial deformations due to head movements and facial expressions [156].”
BackgroundArandjelović, R., & Zisserman, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_27 · Cited by 403
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384
Self-citationNeural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00416 · Cited by 370 · cited 1×
“For example, [190], [191] introduce additional emotion encoders, and [192], [193], [194] devise additional pose encodings into the generation pipeline.”
BackgroundCapture, Learning, and Synthesis of 3D Speaking Styles
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01034 · Cited by 365 · cited 9×
“Based on VOCA, Liu et al. [185] proposed a geometry-guided dense perspective network (GDPnet) with two constraints from different perspectives to achieve a more robust generation.”
MethodInfluentialFlow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00366 · Cited by 355 · cited 3×
“Therefore, Zhang et al. [69] proposed a novel flow-guided VSG framework, including one style-specific animation generator and one flow-guided video generator, to synthesize high visual quality videos.”
BackgroundMethodCombining Residual Networks with LSTMs for Lipreading
Stafylakis, T., & Tzimiropoulos, G. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-85 · Cited by 348 · cited 12×
“Based on C3D-ResNet18 + MSTCN, Ma et al. [53] improved the SOTA to 87.7% on LRW. Recently, more and more works [35], [48], [50], [51], [105], [106] tried to improve visual representations by utilizing extra audio information in the training stage rather than the design of network architectures, as audio signals can pr…”
MethodInfluentialMakeltTalk: speaker-aware talking-head animation
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417774 · Cited by 343
Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., & Wang, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00836 · Cited by 336
Self-citationLearning to Localize Sound Source in Visual Scenes
Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., & Kweon, I. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00458 · Cited by 328
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation
Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., & Loy, C. C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58589-1_42 · Cited by 328 · cited 2×
“As a representative high-quality 4D face audio-visual dataset, VOCASET greatly promoted the research on 3D VSG. MEAD [100], namely Multi-view Emotional Audio-visual Dataset, is a large-scale, high-quality emotional audio-visual dataset.”
BackgroundA deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303 · cited 2×
“7(e), Taylor et al. [67] introduced a simple and effective DNN as a sliding window predictor to automatically learn AAM coefficients based on the fixed-length phoneme sequence.”
BackgroundMethodCooperative Learning of Audio and Video Models from Self-Supervised Synchronization
Korbar, B., Tran, D., & Torresani, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00230 · Cited by 299
Realistic Speech-Driven Facial Animation with GANs
Vougioukas, K., Petridis, S., & Pantic, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01251-8 · Cited by 298 · cited 2×
“For example, some works [40], [163], [196] have observed that both the PSNR and SSIM are negatively affected by introducing image or video discriminators.”
BackgroundMethodEnd-to-End Audiovisual Speech Recognition
Petridis, S., Stafylakis, T., Ma, P., Cai, F., Tzimiropoulos, G., & Pantic, M. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461326 · Cited by 289
Lipreading Using Temporal Convolutional Networks
Martinez, B., Ma, P., Petridis, S., & Pantic, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053841 · Cited by 276 · cited 2×
“To enable temporal backend network capturing multi-scale temporal patterns, Martinez [60] proposed to utilize a multi-scale TCN (MS-TCN) structure, which achieved SOTA results (87.9% Acc) on the word-level LRW dataset.”
MethodInfluentialFaceFormer: Speech-Driven 3D Facial Animation with Transformers
Fan, Y., Lin, Z., Saito, J., Wang, W., & Komura, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01821 · Cited by 263 · cited 1×
“[187] proposed a Transformer-based autoregressive VSG model named FaceFormer to encode the long-term audio context information and predict a sequence of 3D face vertices.”
MethodLRS3-TED: a large-scale dataset for visual speech recognition
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00496 · Cited by 262 · cited 2×
“LRS3-TED [92] is a large-scale sentence-level audio-visual dataset.”
MethodLip Movements Generation at a Glance
Chen, L., Li, Z., Maddox, R. K., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_32 · Cited by 258
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Chen, T. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911195 · Cited by 230 · cited 1×
“Therefore, undoubtedly, visual speech contributes to human speech perception, especially for people who are hearing-impaired or hard of hearing or when acoustic information is corrupted.”
BackgroundMeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Richard, A., Zollhofer, M., Wen, Y., de la Torre, F., & Sheikh, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00121 · Cited by 225 · cited 1×
“[188] proposed a categorical latent space for VSG that disentangles audio-correlated and audio-uncorrelated (facial expressions like eye blinks, eyebrow) information based on a cross-modality loss.”
BackgroundEnd-To-End Audio-Visual Speech Recognition with Conformers
Ma, P., Petridis, S., & Pantic, M. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414567 · Cited by 224
Deep multimodal learning for Audio-Visual Speech Recognition
Mroueh, Y., Marcheret, E., & Goel, V. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178347 · Cited by 216 · cited 2×
“…have improved the number of subjects, dataset scale, recording conditions and script diversity, data quality, etc. Due to the privacy protection laws (e.g., General Data Protection Regulation (GDPR) in Europe Union), some of the existing large-scale datasets [86], [95] are not public available.”
BackgroundRobust speech recognition using articulatory information
Kirchhoff, K. (1998) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 212
Audio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01386 · Cited by 212 · cited 3×
“Referring to the 3DMM based face2face translation pipeline mentioned above, [4, 68, 174, 175, 181, 182, 183] converted the driving source from video to audio clips (text scripts) and migrated this pipeline to VSG tasks.”
BackgroundMethodA no-reference perceptual image sharpness metric based on a cumulative probability of blur detection
Narvekar, N. D., & Karam, L. J. (2009) · 2009 International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2009.5246972 · Cited by 208
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
Xing, J., Xia, M., Zhang, Y., Cun, X., Wang, J., & Wong, T.-T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01229 · Cited by 194
You Said That?: Synthesising Talking Faces from Audio
Jamaludin, A., Chung, J. S., & Zisserman, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01150-y · Cited by 189 · cited 1×
“, Talking Face Generation (TFG) 1 [71, 159].”
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Liu, X., Xu, Y., Wu, Q., Zhou, H., Wu, W., & Zhou, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_7 · Cited by 186
Talking Face Generation by Conditional Recurrent Adversarial Network
Song, Y., Zhu, J., Li, D., Wang, A., & Qi, H. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/129 · Cited by 182
AVICAR: audio-visual speech corpus in a car environment
Lee, B., Hasegawa-Johnson, M., Goudeseune, C., Kamdar, S., Borys, S., Liu, M., & Huang, T. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-424 · Cited by 181
An image transform approach for HMM based automatic lipreading
Potamianos, G., Graf, H. P., & Cosatto, E. (2002) · Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269) · doi:10.1109/icip.1998.999008 · Cited by 180 · cited 1×
“In the era dominated by traditional VSA methods, shallow representations of visual speech such as visemes [17, 18], mouth geometry descriptors [19], linear transformation features [20], statistical representations [21], and sequence modeling like Gaussian process dynamical models [22], hidden Markov models (HMMs) [23]…”
MethodA review of recent advances in visual speech decoding
Zhou, Z., Zhao, G., Hong, X., & Pietikäinen, M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.004 · Cited by 179 · cited 4×
“Several surveys on VSA [43], [44], [45], [46], [47] have been published.”
BackgroundMethodInfluentialSelf-citationAudiovisual Integration and Lipreading Abilities of Older Adults with Normal and Impaired Hearing
Tye-Murray, N., Sommers, M. S., & Spehar, B. (2007) · Ear & Hearing · doi:10.1097/aud.0b013e31812f7185 · Cited by 177
LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
Yang, S., Zhang, Y., Feng, D., Yang, M., Wang, C., Xiao, J., Long, K., Shan, S., & Chen, X. (2019) · 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) · doi:10.1109/fg.2019.8756582 · Cited by 174 · cited 3×
“Meanwhile,theemergence oflarge-scaleVSAdatasets[28],[29],[30],[31],[32]promoted thefurtherdevelopmentofdeeplearningbasedVSAresearch.”
BackgroundMethodGarrido, P., Valgaerts, L., Rehmsen, O., Thormaehlen, T., Perez, P., & Theobalt, C. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.537 · Cited by 172 · cited 1×
“To realize mapping driving sources to lip dynamics, representative traditional VSG methods mainly adopted cross-modal retrieval approaches [18], [109], [151], [152] and HMM-based approaches [153], [154].”
MethodLipNet: End-to-End Sentence-level Lipreading
Assael, Y., Shillingford, B., Whiteson, S., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01599 · Cited by 172 · cited 3×
“Many works [29], [51], [56], [57], [129], [133] have adopted RNN-based (BiLSTM or BiGRU) network architectures as the temporal backend network in VSR.”
BackgroundMethodASR is All You Need: Cross-Modal Distillation for Lip Reading
Afouras, T., Chung, J. S., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054253 · Cited by 170 · cited 2×
“However, some essential advances of VSR were omitted, such as self-supervised learning methods [47, 48, 49, 50], cross-modal knowledge distillation methods [34, 51, 52], graph neural networks backbone architectures [53, 54], etc.”
BackgroundMethodSub-word Level Lip Reading With Visual Attention
Prajwal, K. R., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00510 · Cited by 164
Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis
Shen, S., Li, W., Zhu, Z. D., Duan, Y., Zhou, J., & Lu, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_39 · Cited by 163
Live speech portraits: real-time photorealistic talking-head animation
Lu, Y., Chai, J., & Cao, X. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480484 · Cited by 162
Deep Lip Reading: A Comparison of Models and an Online Application
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1943 · Cited by 157
LCANet: End-to-End Lipreading with Cascaded Attention-CTC
Xu, K., Li, D., Cassimatis, N., & Wang, X. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00088 · Cited by 157
Generative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157
Large-Scale Visual Speech Recognition
Shillingford, B., Assael, Y., Hoffman, M. W., Paine, T., Hughes, C., Prabhu, U., Liao, H., Sak, H., Rao, K., Bennett, L., et al. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1669 · Cited by 154
Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Wang, S., Li, L., Ding, Y., Fan, C., & Yu, X. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/152 · Cited by 153
Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial Decomposition
Tang, J., Wang, K., Zhou, H., Chen, X., He, D., Hu, T., Liu, J., Liu, Z., Zeng, G., & Wang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02481-9 · Cited by 152 · cited 1×
“Motivated by the excellent 3D face structural information modeling ability, more and more works [202], [203], [204], [205] tried to utilize NeRF technology for VSG.”
MethodTowards Practical Lipreading with Distilled and Efficient Models
Ma, P., Martínez, B., Petridis, S., & Pantić, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415063 · Cited by 150
Survey on automatic lip-reading in the era of deep learning
Fernandez-Lopez, A., & Sukno, F. M. (2018) · Image and Vision Computing · doi:10.1016/j.imavis.2018.07.002 · Cited by 148 · cited 4×
“For example, [43, 44, 46] conducted reviews on VSR, and [42, 45] focused on VSG.”
BackgroundMethodInfluentialOuluVS2: A multi-view audiovisual database for non-rigid mouth motion analysis
Anina, I., Zhou, Z., Zhao, G., & Pietikainen, M. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163155 · Cited by 147 · cited 2×
“OuluVS2 [81] ∼2 N/A 2120 53 1920×1080 30 Lab-controlled 5-view facecentered videos 2015 Three types of utterances; High recording quality; Five views from 0-90 [82]”
BackgroundSelf-citationSpeech-Driven Facial Animation Using Cascaded GANs for Learning of Motion and Texture
Das, D., Biswas, S., Sinha, S., & Bhowmick, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_25 · Cited by 141 · cited 2×
“Unlike previous methods using audio MFCC features as input, Sinha et al. [158], [159] introduced DeepSpeech [160] features instead, as DeepSpeech features are more robust for the variation of speakers.”
BackgroundMethodGabbay, A., Shamir, A., & Peleg, S. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1955 · Cited by 140 · cited 1×
“…Togivesomeapplicationexamples,inspeechrecognitionand enhancement,visualspeechcanbetreatedasacomplementary signaltoincreasetheaccuracyandrobustnessofcurrentaudio speechrecognitionandseparationundervariousunfavorable acousticconditions[11],[12].”
Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands
Sun, K., Yu, C., Shi, W., Liu, L., & Shi, Y. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242599 · Cited by 138 · cited 1×
“In human-computer interaction, visual speech can serve as a new type of interactive information, improving the diversity and robustness of interactions [13, 14].”
BackgroundSelf-citationFACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning
Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., & Guo, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00384 · Cited by 137 · cited 1×
“Referring to the 3DMM based face2face translation pipeline mentioned above, [9], [69], [171], [172], [178], [179], [180], [181] converted the driving source from video to audio clips (text scripts) and migrated this pipeline to VSG tasks.”
MethodTowards Automatic Face-to-Face Translation
K R, P., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V., & Jawahar, C. V. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3351066 · Cited by 135 · cited 1×
“For example, Prajwal et al. [39], [195] introduced a simple audio-visual synchronization discriminator for lip-syncing VSG.”
BackgroundRecurrent Neural Network Transducer for Audio-Visual Speech Recognition
Makino, T., Liao, H., Assael, Y., Shillingford, B., Garcia, B., Braga, O., & Siohan, O. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9004036 · Cited by 129 · cited 1×
“[147] ST-CNN RNN-T (BiLSTM) seq2seq loss YT-31khrs √ character 33.”
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 126 · cited 3×
“In the entertainment domain, VSG technology plays a crucial role in personalized 3D talking avatars generation [17] in virtual gaming and realizing high-fidelity photo-realistic talking videos generation for movie post-production like visual dubbing [18].”
BackgroundMethodEverybody’s Talkin’: Let Me Talk as You Want
Song, L., Wu, W., Qian, C., He, R., & Loy, C. C. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3146783 · Cited by 126
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178899 · Cited by 122 · cited 2×
“In2016,deeplearningbasedVSAmethods[27], [28]havevastlyoutperformedtraditionalapproaches,bringing theVSAintothedeeplearningera.”
BackgroundExpressive Talking Head Generation with Granular Audio-Visual Control
Liang, B., Pan, Y., Guo, Z., Zhou, H., Hong, Z., Han, X., Han, J., Liu, J., Ding, E., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00338 · Cited by 119
Vid2speech: Speech reconstruction from silent video
Ephrat, A., & Peleg, S. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953127 · Cited by 114 · cited 1×
“TypicalacademiaandpracticalapplicationsofVSAinclude multimodalspeechrecognitionandenhancement[3],audio-to-videoalignment[4],audiospeechsynthesization[5],[6], speakerrecognitionandverification[7],[8],medicalassistance,…”
Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers
Zhao, Y., Xu, R., Wang, X., Hou, P., Tang, H., & Song, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6174 · Cited by 114
Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading
Zhang, X., Cheng, F., & Shilin, W. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00080 · Cited by 113 · cited 2×
“Due to the considerable performance of the model, plenty of VSR models [51], [53], [60], [62], [128] adopted it as the backbone network for visual features extraction.”
BackgroundMethodOne-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning
Wang, S., Li, L., Ding, Y., & Yu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20154 · Cited by 111
Biometric Liveness Detection: Challenges and Research Opportunities
Akhtar, Z., Micheloni, C., & Foresti, G. L. (2015) · IEEE Security & Privacy · doi:10.1109/msp.2015.116 · Cited by 110 · cited 1×
“In public security, VSA can be applied to face forgery detection [11] and liveness detection [12].”
BackgroundEnd-to-End Speech-Driven Facial Animation with Temporal GANsPreprint
Vougioukas, K., Petridis, S., & Pantić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09313 · Cited by 109 · cited 4×
“For example, Vougioukas et al. [41] adopted the average content distance (ACD) [115] to measuretheaverageeuclideandistanceof target imagerepresen-tation, obtained using OpenFace [116], and the representations of generated frames.”
BackgroundMethodInfluentialImproved Speech Reconstruction from Silent Video
Ephrat, A., Halperin, T., & Peleg, S. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.61 · Cited by 108 · cited 1×
“…multimodalspeechrecognitionandenhancement[3],audio-to-videoalignment[4],audiospeechsynthesization[5],[6], speakerrecognitionandverification[7],[8],medicalassistance,…”
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682524 · Cited by 106
Deep Learning-Based Automated Lip-Reading: A Survey
Fenghour, S., Chen, D., Guo, K., Li, B., & Xiao, P. (2021) · IEEE Access · doi:10.1109/access.2021.3107946 · Cited by 104 · cited 3×
“Several surveys on VSA [43], [44], [45], [46], [47] have been published.”
BackgroundMethodAudio-Visual Recognition of Overlapped Speech for the LRS2 Dataset
Yu, J., Zhang, S.-X., Wu, J., Ghorbani, S., Wu, B., Kang, S., Liu, S., Liu, X., Meng, H. M. L., & Yu, D. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054127 · Cited by 103 · cited 1×
“Someworks[52], [64],[143] 2) Unsupervised Learning for VSR: Unsupervised learning forVSRaimstolearndiscriminativevisualrepresentationswith-outaccesstomanualannotation.”
BackgroundLahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00278 · Cited by 101
Learn an Effective Lip Reading Model without PainsPreprint
Feng, D., Yang, S., Shan, S., & Chen, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07557 · Cited by 99 · cited 2×
“Many works [29], [51], [56], [57], [129], [133] have adopted RNN-based (BiLSTM or BiGRU) network architectures as the temporal backend network in VSR.”
BackgroundMethodSpeech Driven Talking Face Generation From a Single Image and an Emotion Condition
Eskimez, S. E., Zhang, Y., & Duan, Z. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3099900 · Cited by 97
Learning character-agnostic motion for motion retargeting in 2D
Aberman, K., Wu, R., Lischinski, D., Chen, B., & Cohen-Or, D. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322999 · Cited by 93 · cited 1×
“Unlike previous methods, they adopt the cross-reconstruction [185] technique in the audio2expression stage to decompose the input audio into disentangled content and emotion embeddings.”
MethodAnwar, M. A., Shi, B., Goswami, V., Hsu, W.-N., Pino, J., & Wang, C. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2279 · Cited by 89
Speech-Driven 3D Facial Animation with Implicit Emotional Awareness: A Deep Learning Approach
Pham, H. X., Cheung, S., & Pavlovic, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.287 · Cited by 88 · cited 2×
“[171, 172, 176] proposed utilizing CNN + RNN based backbone architectures to map audio signals to blendshape coefficients [177] of a 3D face.”
MethodImitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis
Wu, H., Jia, J., Wang, H., Dou, Y., Duan, C., & Deng, Q. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475280 · Cited by 88 · cited 2×
“Referring to the 3DMM based face2face translation pipeline mentioned above, [4, 68, 174, 175, 181, 182, 183] converted the driving source from video to audio clips (text scripts) and migrated this pipeline to VSG tasks.”
MethodAn audio-visual corpus for multimodal automatic speech recognition
Czyzewski, A., Kostek, B., Bratoszewski, P., Kotus, J., & Szykulski, M. (2017) · Journal of Intelligent Information Systems · doi:10.1007/s10844-016-0438-z · Cited by 87 · cited 2×
“MODALITY [79] ∼31 182 5880 35 1920×1080 100 Lab-controlled Stereoscopic RGB-D facecentered videos 2015 Command-like sentences; High resolution with varying noise conditions [80]”
BackgroundMethodExpressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.434 · Cited by 85 · cited 1×
“…of visual speech such as visemes [19], mouth geometry descriptors [20], linear transformation features [21], statistical representations [22], and sequence modeling like Gaussian process dynamical models [23], hidden Markov models (HMMs) [24], decision tree models [25] 0162-8828 © 2024 IEEE.”
BackgroundSeeing Through Noise: Visually Driven Speaker Separation And Enhancement
Gabbay, A., Ephrat, A., Halperin, T., & Peleg, S. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462527 · Cited by 83
Audiovisual speech synthesis: An overview of the state-of-the-art
Mattheyses, W., & Verhelst, W. (2015) · Speech Communication · doi:10.1016/j.specom.2014.11.001 · Cited by 77 · cited 6×
“Several surveys on VSA [43], [44], [45], [46], [47] have been published.”
BackgroundMethodInfluentialRealistic Mouth-Synching for Speech-Driven Talking Face Using Articulatory Modelling
Xie, L., & Liu, Z.-Q. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.888009 · Cited by 75 · cited 1×
“To realize mapping driving sources to lip dynamics, representative traditional VSG methods mainly adopted cross-modal retrieval approaches [18], [109], [151], [152] and HMM-based approaches [153], [154].”
MethodWhat comprises a good talking-head video generation?: A Survey and BenchmarkPreprint
Chen, L., Cui, G., Kou, Z., Zheng, H., & Xu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03201 · Cited by 75
ObamaNet: Photo-realistic lip-sync from text
Kumar, R., Sotelo, J., Kumar, K., de Brébisson, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01442 · Cited by 74 · cited 2×
“[64] proposed the LSTM + UNet architecture, improving the model by replacing the complex video generation pipeline with a pix2pix framework [161].”
MethodAudio-driven Talking Face Video Generation with Learning-based Personalized Head Pose
Yi, R., Ye, Z., Zhang, J., Bao, H., & Liu, Y.-J. (2020) · arXiv · doi:10.48550/arxiv.2002.10137 · Cited by 73 · cited 3×
“3D Coefficient Based: Besides 2D facial coefficient models, 3D facial coefficients via principal component analysis (PCA) are more commonly used in VSG [68], [71], [168], [169], [170], [171], [172].”
BackgroundMethodAudio-Driven Talking Face Video Generation With Dynamic Convolution Kernels
Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y., Huang, X., Zhang, G., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142387 · Cited by 73 · cited 1×
“7(l), unlike the previous concatenation-based feature fusion strategy, Ye et al. [75] presented a fully convolutional neural network with dynamic convolution kernels (DCKs) for cross-modal feature fusion, which extracts features from audio and reshapes features as DCKs of the fully convolutional…”
BackgroundLiRA: Learning Visual Speech Representations from Audio Through Self-Supervision
Ma, P., Mira, R., Petridis, S., Schuller, B. W., & Pantić, M. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1360 · Cited by 71
A Decision Tree Framework for Spatiotemporal Sequence Prediction
Kim, T., Yue, Y., Taylor, S., & Matthews, I. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783356 · Cited by 70 · cited 1×
“…of visual speech such as visemes [19], mouth geometry descriptors [20], linear transformation features [21], statistical representations [22], and sequence modeling like Gaussian process dynamical models [23], hidden Markov models (HMMs) [24], decision tree models [25] 0162-8828 © 2024 IEEE.”
BackgroundLearning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Ren, S., Du, Y., Lv, J., Han, G., & He, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01312 · Cited by 70 · cited 1×
“Thisiscertainlynotbecauseofalack ofeffortbyresearchers,astherehavebeenmanyexcellentworks onVSA[11],[29],[33],[34],[35],[36].”
Multi-Grained Spatio-temporal Modeling for Lip-readingPreprint
Wang, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11618 · Cited by 67 · cited 4×
“[59]utilized 2DCNNsand3DCNNstoextractbothframe-wisespatialfea-turesandshort-termspatiotemporalfeatures,andthenfusedthe featureswithanadaptivemasktoobtainstrong,multi-grained visualfeatures.”
BackgroundMethodInfluentialWrite-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
Li, L., Wang, S., Zhang, Z., Ding, Y., Zheng, Y., Yu, X., & Fan, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16286 · Cited by 67
Video Rewrite: Driving Visual Speech with Audio
Bregler, C., Covell, M., & Slaney, M. (2023) · Seminal Graphics Papers: Pushing the Boundaries, Volume 2 · doi:10.1145/3596711.3596787 · Cited by 67 · cited 1×
“To realize mapping driving sources to lip dynamics, representative traditional VSG methods mainly adopted cross-modal retrieval approaches [16, 103, 155, 156] and HMM-based approaches [157, 158].”
MethodSpeech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks
Sadoughi, N., & Busso, C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2916031 · Cited by 65
Neural style-preserving visual dubbing
Kim, H., Elgharib, M., Zollhöfer, M., Seidel, H.-P., Beeler, T., Richardt, C., & Theobalt, C. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356500 · Cited by 62 · cited 1×
“Besides the 3D blendshape model, Kim et al. [175], [176] introduced 3D Morphable Model (3DMM) [177], a more dense 3 d face parametric representation, for video-oriented face2face translation.”
MethodAudio/visual mapping with cross-modal hidden Markov models
Fu, S., Gutierrez-Osuna, R., Esposito, A., Kakumanu, P. K., & Garcia, O. N. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.843341 · Cited by 60 · cited 1×
“To realize mapping driving sources to lip dynamics, representative traditional VSG methods mainly adopted cross-modal retrieval approaches [18], [109], [151], [152] and HMM-based approaches [153], [154].”
MethodArbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
Zhu, H., Huang, H., Li, Y., Zheng, A., & He, R. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/327 · Cited by 58 · cited 1×
“Besides the above learning goals, the optical flow discriminator [198], speech-related facial action units [199], and cross-modal mutual information estimator [200] are also utilized to optimize lip motion and cross-modal consistency of generated talking videos with the driving source.”
MethodLipreading with DenseNet and resBi-LSTM
Chen, X., Du, J., & Zhang, H. (2020) · Signal Image and Video Processing · doi:10.1007/s11760-019-01630-1 · Cited by 57 · cited 1×
“Beyond above fundamental RNN structures, various modifications [58, 136] have been made to improve feature learning for VSR.”
BackgroundEnd-to-end Learning for 3D Facial Animation from Speech
Pham, H. X., Wang, Y., & Pavlović, V. (2018) · International Conference on Multimodal Interaction · doi:10.1145/3242969.3243017 · Cited by 56 · cited 2×
“[171, 172, 176] proposed utilizing CNN + RNN based backbone architectures to map audio signals to blendshape coefficients [177] of a 3D face.”
MethodFernandez-Lopez, A., Martínez, O., & Sukno, F. M. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.34 · Cited by 51
Dynamic Temporal Alignment of Speech to Lips
Halperin, T., Ephrat, A., & Peleg, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682863 · Cited by 50
Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading
Sheng, C., Zhu, X., Xu, H., Pietikäinen, M., & Liu, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3102433 · Cited by 49 · cited 5×
“Since the complementarity between image and landmark features, the combination of CNNs and GCNs is often widely adopted [53, 54, 118].”
BackgroundInfluentialSelf-citationChung, S.-W., Kang, H.-G., & Chung, J. S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1113 · Cited by 48
Accurate and Resource-Efficient Lipreading with Efficientnetv2 and Transformers
Koumparoulis, A., & Potamianos, G. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747729 · Cited by 46
Audio-Visual Speech Recognition is Worth $32\times 32\times 8$ Voxels
Serdyuk, D., Braga, O., & Siohan, O. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688191 · Cited by 45
Iterative Text-Based Editing of Talking-Heads Using Neural Retargeting
Yao, X., Fried, O., Fatahalian, K., & Agrawala, M. (2021) · ACM Transactions on Graphics · doi:10.1145/3449063 · Cited by 43
VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices
Kadandale, V. S., Montesinos, J. F., & Haro, G. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10861 · Cited by 39
Geometry-Guided Dense Perspective Network for Speech-Driven Facial Animation
Liu, J., Hui, B., Li, K., Liu, Y., Lai, Y., Zhang, Y., Liu, Y., & Yang, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3107669 · Cited by 38 · cited 1×
“Based on VOCA, Liu et al. [185] proposed a geometry-guided dense perspective network (GDPnet) with two constraints from different perspectives to achieve a more robust generation.”
BackgroundEnd-To-End Generation of Talking Faces from Noisy Speech
Eskimez, Ş. E., Maddox, R. K., Xu, C., & Duan, Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054103 · Cited by 35 · cited 1×
“For temporal-dependent video generation, [41], [196], [197] utilized autoregression style VSG generator networks for talking face generation.”
MethodSpeech-Driven Facial Reenactment Using Conditional Generative Adversarial NetworksPreprint
Jalalifar, S. A., Hasani, H., & Aghajan, H. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.07461 · Cited by 34 · cited 1×
“[165] proposed LSTM + C-GAN VSG backbone architecture, using the basic conditional generative adversarial network (C-GAN) [166] for generating talking faces given the learned landmarks.”
MethodDualLip: A System for Joint Lip Reading and Generation
Chen, W., Tan, X., Xia, Y., Qin, T., Wang, Y., & Liu, T.-Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413623 · Cited by 34 · cited 3×
“Thisiscertainlynotbecauseofalack ofeffortbyresearchers,astherehavebeenmanyexcellentworks onVSA[11],[29],[33],[34],[35],[36].”
BackgroundMethodTalking Face Generation with Expression-Tailored Generative Adversarial Network
Zeng, D., Liu, H., Lin, H. Z., & Ge, S. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413844 · Cited by 34 · cited 1×
“Besides the above learning goals, the optical flow discriminator [198], speech-related facial action units [199], and cross-modal mutual information estimator [200] are also utilized to optimize lip motion and cross-modal consistency of generated talking videos with the driving source.”
MethodHAVRUS Corpus: High-Speed Recordings of Audio-Visual Russian Speech
Verkhodanova, V., Ronzhin, A., Kipyatkova, I. S., Ivanko, D., Karpov, A. A., & Železný, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-43958-7_40 · Cited by 33
Speech2Talking-Face: Inferring and Driving a Face with Synchronized Audio-Visual Representation
Sun, Y., Zhou, H., Liu, Z., & Koike, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/141 · Cited by 33 · cited 1×
“Sun et al. [73] improved the model by learning speech and identity features within a self-supervised contrastive learning framework, with no need for extra annotations.”
Identity-Preserving Realistic Talking Face Generation
Sinha, S., Biswas, S., & Bhowmick, B. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206665 · Cited by 32 · cited 2×
“Unlike previous methods using audio MFCC features as input, Sinha et al. [158], [159] introduced DeepSpeech [160] features instead, as DeepSpeech features are more robust for the variation of speakers.”
BackgroundMethodPerfect Match: Self-Supervised Embeddings for Cross-Modal Retrieval
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987720 · Cited by 30 · cited 4×
“Many deep learning driven technologies [39], [105], [106], [107] have been introduced to address the alignment of A-V signals in a self-supervised way, i.e., automatically creating positive and negative A-V sync/out-of-sync pairs.”
BackgroundInfluentialImproving Audio-visual Speech Recognition Performance with Cross-modal Student-teacher Training
Li, W., Wang, S., Lei, M., Siniscalchi, S. M., & Lee, C. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682868 · Cited by 26 · cited 2×
“KD based: self-KD [52], cross-modal KD [63]”
MethodDeena, S., Hou, S., & Galata, A. (2010) · ICMI-MLMI '10 · doi:10.1145/1891903.1891942 · Cited by 23 · cited 1×
“In the era dominated by traditional VSA methods, shallow representations of visual speech such as visemes [17, 18], mouth geometry descriptors [19], linear transformation features [20], statistical representations [21], and sequence modeling like Gaussian process dynamical models [22], hidden Markov models (HMMs) [23]…”
MethodExploring ROI size in deep learning based lipreading
Koumparoulis, A., Potamianos, G., Mroueh, Y., & Rennie, S. J. (2017) · The 14th International Conference on Auditory-Visual Speech Processing · doi:10.21437/avsp.2017-13 · Cited by 21 · cited 1×
“[117] proved that the selection of ROI will significantly affect the final recognition performance, but it is still unable to determine the optimal ROI.”
BackgroundMODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions
Liu, Y., Lin, L., Yu, F., Zhou, C., & Li, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02104 · Cited by 20 · cited 1×
“Besides 2D landmark based approaches, mapping driving sources to 3D landmarks is also widely explored [17], [165], [166].”
BackgroundOne-shot Talking Face Generation from Single-speaker Audio-Visual Correlation LearningPreprint
Wang, S., Li, L., Ding, Y., & Yu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02749 · Cited by 20 · cited 2×
“They recently updated the motion field generator by replacing the 3D Hourglass Network with a self-attention architecture [42].”
BackgroundMethodTalking Head Generation with Audio and Speech Related Facial Action Units
Chen, S., Liu, Z., Liu, J., Yan, Z., & Wang, L. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.78 · Cited by 20
Sheng, C., Pietikäinen, M., Tian, Q., & Liu, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475415 · Cited by 19 · cited 8×
“A potential research direction is to realize crossmodal self-supervised visual speech learning [49, 50, 101] based on unlabeled audio-visual data.”
BackgroundMethodInfluentialSelf-citationLip Graph Assisted Audio-Visual Speech Recognition Using Bidirectional Synchronous Fusion
Liu, H., Chen, Z., & Yang, B. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3146 · Cited by 18 · cited 6×
“[53] proposed the first end-to-end GCN model (ST-GCN) that extracts shapebased visual features by learning the lip landmark points and their relationships.”
BackgroundInfluentialReal-time Neural Radiance Talking Portrait Synthesis via Audio-spatial DecompositionPreprint
Tang, J., Wang, K., Zhou, H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12368 · Cited by 17 · cited 1×
“Motivated by the excellent 3D face structural information modeling ability, more and more works [202], [203], [204], [205] tried to utilize NeRF technology for VSG.”
MethodCross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation
Yang, C.-C., Fan, W.-C., Yang, C.-F., & Wang, Y.-C. F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20210 · Cited by 16 · cited 5×
“This is certainly not because of a lack of effort by researchers, as there have been many excellent works on VSA [6, 28, 32, 33, 34, 35].”
BackgroundInfluentialSynthesising 3D Facial Motion from “In-the-Wild” Speech
Tzirakis, P., Papaioannou, A., Lattas, A., Tarasiou, M., Schuller, B. W., & Zafeiriou, S. P. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00100 · Cited by 13 · cited 1×
“Pham et al. [168], [169], [173] proposed utilizing CNN + RNN based backbone architectures to map audio signals to blendshape coefficients [174] of a 3D face.”
MethodSpeech2Video: Cross-Modal Distillation for Speech to Video Generation
Si, S., Wang, J., Qu, X., Cheng, N., Wei, W., Zhu, X., & Xiao, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1996 · Cited by 12
Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models
Abdelaziz, A. H., Theobald, B.-J., Binder, J. G., Fanelli, G., Dixon, P. R., Apostoloff, N., Weise, T., & Kajareker, S. (2019) · International Conference on Multimodal Interaction · doi:10.1145/3340555.3353745 · Cited by 11 · cited 2×
“[173] finetuned a pretrained DNN-based acoustic model to map driving audios to 3D blendshape coefficients, as they hold the idea that a pretrained acoustic model has better generalization on speaker-independent VSG tasks than a randomly initialized model.”
MethodMobiLipNet: Resource-Efficient Deep Learning Based Lipreading
Koumparoulis, A., & Potamianos, G. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2618 · Cited by 11 · cited 3×
“Besides VGG and ResNet, researchers have also adopted other representative 2D CNN architectures, including DenseNet [58], ShuffleNet [52], MobileNet [129], etc.”
MethodLip Reading using Local-Adjacent Feature Extractor and Multi-Level Feature Fusion
Zhang, C., & Zhao, H. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1883/1/012083 · Cited by 9 · cited 2×
“On the other hand, as landmark points are irregular data, some existing works [53, 54, 118] adopted Graph Convolution Networks (GCNs) to extract visual features from landmark points.”
Background(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1 · cited 1×
“Unlike previous methods using audio MFCC features as input, Sinha et al. [158], [159] introduced DeepSpeech [160] features instead, as DeepSpeech features are more robust for the variation of speakers.”
MethodVoxCeleb: a large-scale speaker identification dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · arXiv · doi:10.48550/arxiv.1706.08612 · cited 2×
“VoxCeleb1 [30] is a large-scale text-independent audio-visual dataset collected from open-source YouTube media.”
BackgroundVoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1806.05622 · cited 2×
“Meanwhile, the emergence of large-scale VSA datasets [27, 28, 29, 30, 31] promoted the further development of deep learning based VSA research.”
BackgroundGabbay, A., Shamir, A., & Peleg, S. (2017) · arXiv · doi:10.48550/arxiv.1711.08789
Seeing Through Noise: Visually Driven Speaker Separation and Enhancement
Gabbay, A., Ephrat, A., Halperin, T., & Peleg, S. (2017) · arXiv · doi:10.48550/arxiv.1708.06767 · cited 1×
“…multimodalspeechrecognitionandenhancement[3],audio-to-videoalignment[4],audiospeechsynthesization[5],[6], speakerrecognitionandverification[7],[8],medicalassistance, security,forensic,videocompression,entertainment,human-computerinteraction,emotionunderstanding[9],[10],etc.…”
Research works printed but not linked: 6
- He, Dual learning for machine translation, 820
- Goodfellow, Generative adversarial nets, 2672
- Weng, Learning spatio temporal features with two stream deep 3D CNNs for lipreading
- Chung, You said that?
- 10.1609/aaai.v36i3.20154 · DOI · Paper page
- Ye, GeneFace: Generalized and high-fidelity audio-driven 3D talking face synthesis
Cited by
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Peng, Z., Wu, H., Song, Z., Xu, H., Zhu, X., He, J., Liu, H., & Fan, Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01891 · Cited by 159
“However, current methods mainly focus on improving the synchronization between lip movements and speech [38], neglecting the emotional variation of facial expressions.”
BackgroundXing, Z., Ma, G., Wang, L., Yang, L., Guo, X., & Chen, S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3546874 · Cited by 98
“Compared to manually designed features, deep features have stronger and more robust representation capabilities [46], [47], [48].”
Talking human face generation: A survey
Toshpulatov, M. A., Lee, W., & Lee, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119678 · Cited by 47
CNN Learning Strategy for Recognizing Facial Expressions
Lee, D., & Yoo, J. (2023) · IEEE Access · doi:10.1109/access.2023.3294099 · Cited by 40
“Despite numerous studies aimed at improving the CNN performance, training the CNN model with data collected under various conditions remains a challenge in several domains, such as object detection [36], activity recognition [37], and visual speech recognition [38].”
BackgroundA Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 40
“It also enables end-to-end learning by mapping the source data into the expected target without the need for handcrafted features as was the case with prior techniques [5], [17].”
BackgroundMethod
Metadata
- Authors
- Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L.
- Affiliations
- National University of Defense Technology (CN)
- University of Oulu (FI)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-09
- Volume
- 46
- Issue
- 9
- Pages
- 6001-6022
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4392780157
- Cited by
- 64
- References
- 187
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 18
- FWCI
- 5.05
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2021YFB3100800)
- Academy of Finland (331883)
- National Natural Science Foundation of China (62376283, 61872379)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- visual speech, visual speech recognition, deep learning, benchmark dataset, research challenges
- MeSH terms
- Deep Learning, Humans, Image Processing, Computer-Assisted, Speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.