Deep Learning for Visual Speech Analysis: A Survey
Open accessThe DOI matches its Crossref record; 188 of the 189 references it cites were found in a registry or an index, and 1 are not checked yet; 10 of the 273 sources its claims cite were checked for support.
Abstract
Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.
Verified 2026-10-11 · 1 reference not checked yet
Overview
CoChat Summary
## TL;DR
This survey reviews deep learning methods for visual speech analysis, focusing on the two dual tasks of visual speech recognition (VSR) and visual speech generation (VSG). It summarizes datasets, challenges, evaluation metrics, representative methods, and reported state-of-the-art results, and it highlights open problems and future directions.
## Key findings
- The paper frames VSA around two formally dual problems: VSR and VSG, and argues that VSR and VSG share core technical points such as visual speech representation learning and contextual sequence modeling.
- For VSR, the paper reports strong recent performance: the SOTA on LRW reached 88.5% and on LRW-1000 50.5% with cross-modal mutual learning; on the simple GRID dataset, deep VSR models almost saturate at 1.3% WER.
- For word-level VSR, a baseline C3D-ResNet34 + BiLSTM achieved 83.5% on LRW, and later methods improved to 87.7% on LRW; MS-TCN-based methods are reported at 85.3% on LRW and 41.4% on LRW-1000.
- For VSG, the paper compares representative methods using PSNR, SSIM, and LMD, and reports examples such as 29.36 PSNR / 0.74 SSIM / 1.35 LMD for a one-stage method on VoxCeleb+LRW, and 33.4 PSNR / 0.96 SSIM / 0.60 LMD for Prajwal et al. on LRS2.
## Methods
- Survey design: a comprehensive literature review organized around two main tasks, VSR and VSG, plus datasets, evaluation metrics, and practical challenges.
- Coverage: discusses benchmark datasets in controlled and uncontrolled settings, with examples including AVICAR, GRID, OuluVS2, LRW (> 1,000k word instances), LRS2-BBC (> 144.5k utterances), VoxCeleb1 (> 100k utterances from 1,251 celebrities), LRS3-TED (> 400 hours), and MuAViC (1,200 hours in 9 languages).
- Analysis: synthesizes methods by taxonomy and reports representative performance comparisons from published results rather than conducting new experiments.
## Limitations
- The paper explicitly says that many practical issues remain unresolved, including the need for fewer labels, better benchmark fairness, improved robustness analysis, and better evaluation benchmarks for VSG.
- It states that fair comparison is difficult because VSR/VSG results are affected by extra language models, extra training datasets, audio signals, implementation details, and non-unified test platforms.
- Evident from the text: this is a survey, so it does not present a new experimental dataset, model, or unified benchmark of its own; instead it depends on heterogeneous reported results from prior work.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 50 Not valid · 176 Not checked · 132 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Meanwhile, the emergence of large-scale VSA datasets [28, 29, 30, 31, 32] promoted the further development of deep learning based VSA research.”
What the sources say
- Not checked yetcites [28] matched to “Lip Reading in the Wild” (2017) source
- Not checked yetcites [29] source
- Supportedcites [30] matched to “VoxCeleb2: Deep Speaker Recognition” (2018) source
The Introduction (p. 1) presents VoxCeleb2 as a large-scale speaker-recognition dataset addressing a shortage of such data. The paper also reports training a deep CNN speaker-recognition system on it and achieving state-of-the-art verification performance (p. 1; Conclusion, p. 4). This supports its use as an example of a large-scale dataset advancing deep-learning-based speaker recognition. (LLM, conf=0.90)
- Supportedcites [31] matched to “VoxCeleb: A Large-Scale Speaker Identification Dataset” (2017) source
The Abstract identifies VoxCeleb as a large-scale speaker dataset and reports CNN-based identification and verification results on it. Related Works says a large real-world dataset is essential to the shift toward deep CNNs for speaker identification, supporting the cited connection between such datasets and deep-learning research. (LLM, conf=0.91)
- Not checked yetcites [32] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Many deep learning driven technologies [39, 105, 106, 107] have been introduced to address the alignment of AV signals in a self-supervised way, i . e ., automatically creating positive and negative A-V sync/out-of-sync pairs.”
What the sources say
- Supportedcites [39] matched to “A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild” (2020) source
Section 3.3.1 describes an audio-video sync discriminator trained with automatically sampled in-sync and out-of-sync pairs: an audio window is taken either from the matching time or a different time. Section 3.3.2 says the paper adapts and trains that discriminator for its own lip-generation method, supporting its use as an example of the technology described. (LLM, conf=0.90)
- Not checked yetcites [105] source
- Not checked yetcites [106] source
- Not checked yetcites [107] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“[11] is the first work introducing transformers to VSR.”
What the sources say
- Partly supportedcites [11] matched to “Deep Audio-Visual Speech Recognition” (2022) sourceExperimental — accuracy not yet measured
Section 3 (page unavailable) describes Transformer-based audio-visual speech-recognition models, supporting the claim that this work uses Transformers for VSR. But the Introduction (page unavailable) says one Transformer variant had already been published in [2]; our copy does not establish that this paper was the first to introduce Transformers to VSR. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
What the sources say
- Partly supportedcites [11] matched to “Deep Audio-Visual Speech Recognition” (2022) sourceExperimental — accuracy not yet measured
The Introduction and §6.1 (Lips only; page unavailable) report that the Transformer-based model improved sentence-level visual speech recognition from 70.4% to 48.3% word error rate relative to the earlier model. But the Introduction says the visual frontend was also updated and attributes the improvement to both frontend and backend changes; the experiments do not isolate a Transformer-versus-RNN backend effect. (LLM, conf=0.87)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Except for marginbased loss, L1 loss and binary classification loss [48, 149, 150, 151] are also widely used for A-V representations learning.”
What the sources say
- Not checked yetcites [48] source
- Partly supportedcites [149] matched to “Learning to Localize Sound Source in Visual Scenes” (2018) sourceExperimental — accuracy not yet measured
Our copy of Senocak et al. (CVPR 2018) supports only the binary-classification half of the claim: in Sec. 4 (Eq. 7) the supervised loss L_S is a cross-entropy over the attention map whose ground-truth values α_GT,i are binary, and the paper notes it performs slightly better than the mean-square-error alternative. The L1-loss part is not in our copy — the unsupervised objective is the distance-ratio/triplet loss (Eq. 5, Sec. 4) built on ℓ2 distances, and no ℓ1 loss appears anywhere (searched: L1 loss, distance/embedding objective, loss ablation). The paper also makes no statement that either loss is 'widely used' for audio-visual representation learning; it is a single method paper, so the survey-level framing rests on the other citations in the cluster. (LLM, conf=0.68)
- Not checked yetcites [150] source
- Not checked yetcites [151] matched to “Video Rewrite: driving visual speech with audio” (1997) source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
97 more claims · 1 with their cited sources checked
Create a free account to see all 367 claims and to verify the 263 references still open.
- Probably not a substantial claim. 216
- It may state two separate claims as one. 106
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1.2 Datasets under uncontrolled environments
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
§ 3.1.2 Datasets under uncontrolled environments
“Compared to LRS2-TED, it has a larger scale in terms of duration, vocabulary, and number of speakers.”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Representative CNN architectures, such as VGG [124], ResNet [125], MobileNet [126], DenseNet [127] etc, have been widely used in learning visual representation for VSR.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformer-Based Architectures: Compared to RNN-based architectures, Transformers [136] have significant advantages in long-term dependency and parallel computation.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 5×
“Representative CNN architectures, such as VGG [121], ResNet [122], MobileNet [123], DenseNet [124], ShuffleNet [125] etc, have been widely used in learning visual representation for VSR.”
MethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Since AlexNet [26] was proposed in 2012, researchers have invented a variety of deeper, wider, and Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“[64] proposed the LSTM + UNet architecture, improving the model by replacing the complex video generation pipeline with a pix2pix framework [161].”
MethodMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Research works printed but not linked: 6
- He, Dual learning for machine translation, 820
- Goodfellow, Generative adversarial nets, 2672
- Weng, Learning spatio temporal features with two stream deep 3D CNNs for lipreading
- Chung, You said that?
- 10.1609/aaai.v36i3.20154 · DOI · Paper page
- Ye, GeneFace: Generalized and high-fidelity audio-driven 3D talking face synthesis
Cited by
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Peng, Z., Wu, H., Song, Z., Xu, H., Zhu, X., He, J., Liu, H., & Fan, Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01891 · Cited by 159
“However, current methods mainly focus on improving the synchronization between lip movements and speech [38], neglecting the emotional variation of facial expressions.”
BackgroundXing, Z., Ma, G., Wang, L., Yang, L., Guo, X., & Chen, S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3546874 · Cited by 98
“Compared to manually designed features, deep features have stronger and more robust representation capabilities [46], [47], [48].”
Talking human face generation: A survey
Toshpulatov, M. A., Lee, W., & Lee, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119678 · Cited by 47
CNN Learning Strategy for Recognizing Facial Expressions
Lee, D., & Yoo, J. (2023) · IEEE Access · doi:10.1109/access.2023.3294099 · Cited by 40
“Despite numerous studies aimed at improving the CNN performance, training the CNN model with data collected under various conditions remains a challenge in several domains, such as object detection [36], activity recognition [37], and visual speech recognition [38].”
BackgroundA Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 40
“It also enables end-to-end learning by mapping the source data into the expected target without the need for handcrafted features as was the case with prior techniques [5], [17].”
BackgroundMethodLFIR-YOLO: Lightweight Model for Infrared Vehicle and Pedestrian Detection
Wang, Q., Liu, F., Cao, Y., Ullah, F., & Zhou, M. (2024) · Sensors · doi:10.3390/s24206609 · Cited by 27
Rahim, M. A., Rahman, M. M., Islam, M. S., Muzahid, A. J. M., Rahman, M. A., & Ramasamy, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125080 · Cited by 30
Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques
Seegehalli, P. J., & Niranjana Krupa, B. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103738 · Cited by 20
Jeon, S., Lee, J., Yeo, D., Lee, Y., & Kim, S. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0266 · Cited by 13
A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track
Alshahrani, M., & Maashi, M. (2024) · IEEE Access · doi:10.1109/access.2024.3404056 · Cited by 9
“Eleven studies have stated English as the primary spoken language for synchronization [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [18], [19], [20], [21], [22], [24], [30].”
BackgroundLearn2Talk: 3D Talking Face Learns From 2D Talking Face
Zhuang, Y., Cheng, B., Cheng, Y., Jin, Y., Liu, R., Li, C., Cheng, X., Liao, J., & Lin, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3476275 · Cited by 24
“Fromthemethod-ologicalperspective,themethodsareroughlycategorizedinto twoclasses[1]:one-stageframeworkandtwo-stageframework.”
Urfan, M., Rajput, P., Mahajan, P., Sharma, S., Hakla, H. R., Kour, V., Khajuria, B., Chowdhary, R., Lehana, P. K., Karlupia, N., et al. (2024) · Research · doi:10.34133/research.0491 · Cited by 11
“With the ever-increasing human population, early and precise detection of crop systems prone to or suffering from pest diseases and nutrient deficiencies is of utmost importance [ 11 ].”
BackgroundMethodDjilali, Y. A. D., Narayan, S., Boussaid, H., Almazrouei, E., & Debbah, M. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01268 · Cited by 24
“Whilst we briefly overview the recent sentence-level VSR efforts, we refer to [51, 63, 18] for extensive reviews.”
BackgroundSpatiotemporal Feature Enhancement for Lip-Reading: A Survey
Ma, Y., & Sun, X. N. (2025) · Applied Sciences · doi:10.3390/app15084142 · Cited by 10
Deep Transfer Learning for Lip Reading Based on NASNetMobile Pretrained Model in Wild Dataset
Ameer, A. W. A., Salehpour, P., & Asadpour, M. (2024) · IEEE Access · doi:10.1109/access.2024.3470521 · Cited by 8
“In wild datasets, domain diversity issues such as limited variations in language, speaker characteristics, and accents can bias deep learning model training [31].”
When deep learning deciphers silent video: a survey on automatic deep lip reading
Oghbaie, M., Sabaghi, A., Hashemifard, K., & Akbari, M. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20156-4 · Cited by 11
Pose-Aware 3D Talking Face Synthesis Using Geometry-Guided Audio-Vertices Attention
Li, B., Wei, X., Liu, B., He, Z., Cao, J., & Lai, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3371064 · Cited by 13
“As a key technology of human-computer interaction in the virtual environment, audio-driven facial animation has attracted a lot of research [1], [6].”
DiffDub: Person-Generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-Encoder
Liu, T., Du, C., Fan, S., Chen, F., & Yu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446049 · Cited by 9
“The task navigates through multiple challenges[5]: maintaining high visual quality , ensuring temporal consistency , and perfecting lip synchronization .”
BackgroundDeep Learning-Based Oral Cancer Detection through Speech Analysis
Cherukupalli, M., Ragavan, B., Pooja, S., Jaideep, S., & Supriya, P. (2024) · Control Instrumentation System Conference (CISCON) · doi:10.1109/ciscon62171.2024.10695997 · Cited by 9
“Study [19] provides a comprehensive review of recent progress in deep learning methods for visual speech analysis.”
MethodA Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection
Lee, K., Zhang, Y., & Duan, Z. (2024) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp61759.2024.10743671 · Cited by 15
“The speech could be generated from text-to-speech [2] and voice conversion algorithms [3], [4], and the videos are either face-swap [5] from an original video or further rendered from speech and a still image [6].”
BackgroundZhou, D., Zhang, Y., Wu, J., Zhang, X., Xie, L., & Yin, E. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3585165 · Cited by 16
“SSR primarily encompasses visual speech recognition [23] and bio-signal-based speech recognition [10].”
Multi-modal co-learning for silent speech recognition based on ultrasound tongue images
Guo, M., Wei, J., Zhang, R., Zhao, Y., & Fang, Q. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103140 · Cited by 12
Wei, X., Yu, W., Liu, R., Ma, M., Zhao, C., & Shen, G. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3601673 · Cited by 6
A review of generative models for virtual human motion driving
Wang, P., Peng, H., Mao, S., Wang, S., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-023-4284-x · Cited by 5
Physics-Informed Neural Networks for Enhanced Thermal Regulation in a Spacecraft
de Curtò, J., & de Zarz, I. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.647 · Cited by 4
Fausto, I. R. D. S., Braz, R. M. M., & Leta, F. R. (2024) · Brazilian Journal of Development · doi:10.34117/bjdv10n6-055 · Cited by 3
Conformal deep forest for uncertainty-aware classification
Zhang, J., Qiu, Y., & Dong, L. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00175-3 · Cited by 3
“…models have made significant advances across a broad spectrum of domains and applications, including (but not limited to) visual speech analysis (Sheng et al. 2024), natural language processing (Otter et al. 2021), autonomous driving (Ma et al. 2024), pollutant prediction (Liu et al. 2024),…”
Dual-Stream Former: A Dual-Branch Transformer Architecture for Visual Speech Recognition
Jeon, S., Lee, J., & Lee, Y. (2025) · AI · doi:10.3390/ai6090222 · Cited by 3
Wang, L., He, H., Dong, Y., Li, X., Gan, W., & Zhang, X. (2025) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2025.104456 · Cited by 3
DMFP: Dynamic multiscale feature perturbations for transferable adversarial attacks
Cheng, S., Li, P., Han, K., Zheng, Y., Xu, H., & Yao, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114469 · Cited by 3
Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration
Rathod, R. A. M. S. B. (2024) · Journal of Electrical Systems · doi:10.52783/jes.1367 · Cited by 3
“[4] We also outline the objectives of our proposed approach and the structure of the paper.”
MethodWu, J., An, X., Zhang, Y., Zheng, C., Zhang, X., Xie, L., & Yin, E. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107783 · Cited by 2
Lips Reading Spoken Arabic Word Based on The Geometric Shape Features of The Lip
Alsaif, K., & AL-lella, N. S. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2310164 · Cited by 2
Learning of Molecular Graphs in Toxicity Prediction
Huang, Y., Peng, D., & Wang, D. D. (2024) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc63072.2024.10935232 · Cited by 1
“Having boosted a wide range of image and speech tasks [12, 13, 14, 15, 16], deep learning becomes more and more prevalent in many application fields.”
Fu, M., Wang, X., Wang, J., & Zhang, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3574567 · Cited by 1
“While deep learning methods have demonstrated e ffi cacy across various recognition tasks [4], [5], [6], [7], [8], [9], [10], their reliance on extensive training data often poses challenges in low-resource settings.”
Li, H., Cao, H., Cui, J., & Geng, J. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3602278 · Cited by 1
“These modalities exhibit considerable complementarity [6], [7], [8], [9], [10].”
OXSeg: Multidimensional Attention UNet-Based Lip Segmentation Using Semi-Supervised Lip Contours
Moghaddasi, H., Chambers, C., Mattson, S. N., Wozniak, J. R., Coles, C. D., Mukherjee, R., & Suttie, M. (2025) · IEEE Access · doi:10.1109/access.2025.3577000 · Cited by 1
“…segmentation has a broad application across several domains, including cosmetics, e.g., improving lip wrinkles [4], speech recognition tasks such as lip synchronization [5], and automatic lip-reading (or visual speech recognition) [6] , and the diagnosis of facially affected conditions [7], [8].”
TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading
Lee, B. H., Shin, W., & Han, S. W. (2025) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2025.104540 · Cited by 3
“Most existing lipreading studies have primarily focused on improving recognition accuracy, often over-looking the practical constraints associated with real-time deployment [63].”
Design and implementation of metaheuristic-driven BiLSTM hybrid model for silent communication
Sonawane, S., & Malathi, P. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107914 · Cited by 1
AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition
Xue, J., Liu, X., Wu, X., Yin, X., Huang, D., & Yu, F. (2025) · doi:10.1145/3728425.3759905 · Cited by 3
“However, relying solely on audio can be limiting, especially when the signal is noisy or missing [30].”
A Survey of Talking-Head Generation Technology and Its Applications
Han, L., Zhang, L., Zou, X., Lin, J., Li, X., Li, Y., Chang, X., & Zheng, J. (2025) · International Conference on Electrical Automation and Artificial Intelligence (ICEAAI) · doi:10.1109/iceaai64185.2025.10956370 · Cited by 1
“Early methods for head-talking video generation primarily relied on cross-modal retrieval [6] and Hidden Markov Model (HMM) based approaches [7] to map driving sources to lip movement data.”
Nadeem, A., Khan, M. Z., Sirshar, M., & Thirunavukkarasu, U. (2026) · Diagnostics · doi:10.3390/diagnostics16152377
“Traditional diagnostic techniques in the field usually depend on the subjective observation and listening of a speech–language therapist, often being subjective, time-intensive, and expert-dependent [3].”
Assessing Visual Speech Communication Abilities: Cross-Situational Consistency in Lipreadability
Myerson, J., Spehar, B., Strube, M., Hale, S., Tye‐Murray, N., Sommers, M. S., & Peelle, J. E. (2026) · Multisensory Research · doi:10.1163/22134808-bja10205
Optimizing Artificial Neural Networks Trough Weight Adjustments
Akber, S. M. A., Szczęsna, A., & Kazmi, S. N. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.615
A Lightweight and High-Fidelity Model for Generalized Audio-Driven 3D Talking Face Synthesis
Liu, S., Zhong, Y., Wang, H., & Peng, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78125-4_6
Transfer Learning for Limited-Data Infra-Red Lipreading Model Training
Daou, S., Hamadou, A. B., Rekik, A., & Kallel, A. (2024) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa63423.2024.10912550
“These include the variability in lip shapes and movements across individuals, the influence of facial pose, the presence of homophones, and varying lighting conditions [5].”
Wang, X., Jiang, B., & Li, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9949-0_2
Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language
Pham, D. N., & Avramidis, E. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099322
“It serves to evaluate the models’ robustness - how well they perform when faced with unseen data under unexpected or adversarial conditions.”
Audio visual chinese speech recognition in daily scene based on cross modal context encoder
Liu, Y., Qu, Z., Li, J., Zhang, Q., Liu, B., & Chen, W. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04788-z
EventLip: Enhancing Event-Based Lip Reading via Frequency-Aware Spatiotemporal Hypergraph Modeling
Zhang, X., Sun, J., Zhang, C., Yue, X., Xiao, T., Cai, S., Lao, M., & Li, H. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755404 · Cited by 3
“Automatic Lip Reading (ALR) [11, 48, 72, 74, 75], as an important complement to auditory speech recognition, has demonstrated great potential in challenging scenarios where the speech signal is unclear or heavily degraded [11, 28, 62, 69].”
EmoRLTalk: Speech-Driven Emotional Facial Animation With Offline Reinforcement Learning
Liu, G., Li, X., Gao, R., Yuan, Y., Liu, J., Li, H., Huo, H., & Fang, T. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246337
“However, existing methods still face multiple challenges in achieving accurate alignment between emotional expression and speech content, maintaining lip-sync accuracy, and providing user controllability [12].”
Semantics-aware high-frequency enhancement for event-based lip-reading
Zhang, Y., Liu, X., Zhou, J., Liu, Y., & Tan, H. (2025) · Information Sciences · doi:10.1016/j.ins.2025.123026 · Cited by 2
Ain, Q. U., Afzal, H., Subhan, F., Su’ud, M. M., & Jung, Y. (2026) · Big Data · doi:10.1177/2167647x251411174
“9 – 13 Inspired by these achievements, 14 – 19 this study leverages state-of-the-art deep learning architectures to address the pressing challenge of dysarthric speech recognition, aiming to bring similar breakthroughs to the domain of impaired speech processing.”
DeepLip: A Hybrid CNN-RNN-LSTM Framework for End-to- End Visual Speech Recognition using CTC Loss
Verma, A. (2026) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2026.77475
Challenges and enhancements in Turkish automatic lip reading using deep learning models
Sabaz, F., Atila, Ü., Dörterler, M., & Uçan, A. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-026-05252-2
JUND-F0: A Novel Deep Learning Framework for Joint Unvoiced/Voiced Detection And F0 Estimation
Chen, Y., Feng, R., Liu, Y.-L., Hu, Y., & Yuan, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464568
Verma, A., Buragohain, R., Kumar, A., & Nathwani, K. (2026) · National Conference on Communications · doi:10.1109/ncc68160.2026.11479142
“VMD has demonstrated efficacy in various biomedical applications, including EEG seizure identification [22], cardiac arrhythmia classification [23], and electromyographic gesture recognition [24], yet it remains unexplored for oral cancer speech characterization.”
Takamichi, S., Nakamura, T., Suda, H., Fukayama, S., & Ogata, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463396
Ain, Q. U., Afzal, H., Islam, I. U., Khan, J., Jung, Y., & Sivakumar, N. R. (2026) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2026.3689688
“These data-driven approaches allow computational models to learn meaningful representations from speech signals, identify subtle acoustic variations, and perform accurate predictions without explicit rule-based programming [16], [17], [18], [19], [20], [21].”
Goyal, R., Verma, L., & Gupta, T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19185-4_2
Let Toon Talk: Speech-Driven 3D Cartoon Animation via Parametric Modeling and Flow Matching
Wang, D., Cao, S., & Tang, B. (2026) · Applied Sciences · doi:10.3390/app16104840
Du, J., (6206297), J. W., Chen, W., Liu, P., & Wang, T. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109207
A Multimodal Preprocessing Pipeline for Robust Audio-Visual Speech Separation and Recognition
Sh, S. M., & Albaker, B. M. (2026) · HighTech and Innovation Journal · doi:10.28991/hij-2026-07-02-023
Quoc, T. N., Thien, P. N. T., Hoang, V. T., & Tran-Trung, K. (2026) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn70047.2026.11594266
“Recent surveys further highlight that data protocol, preprocessing choices, and evaluation setup remain as important as backbone selection, especially in low-resource or under-documented settings [2], [17].”
From deep learning to large language models: A review of visual speech recognition
Wang, X., He, X., Jiang, W., Ding, X., Meng, X., Du, T., Ou, Y., Shan, R., Shibiao, S., Zhu, H., et al. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200704
Mudra: A Visual Speech Recognition System Using CNN-LSTM Architecture
Reshma, J., S, G. G., Reddy, D. S., Kamath, P. P., & Prakruthi, S. (2026) · IEEE Nexo-Tech International Conference on Advanced Technologies and Innovations (ICATI) · doi:10.1109/icati69399.2026.11604237
“Sheng et al. [2] performed an enormous survey that pit CNNs, RNNs, and the new Transformer architectures against each other for the field of visual speech analysis.”
Lightweight Restricted Dialogue Visual Speech Recognition for Embedded Systems
Moemeng, C., Thumthawatworn, T., Channarukul, S., & Charoenvikrom, S. (2026) · International Conference on Electronics Technology (ICET) · doi:10.1109/icet69987.2026.11659293
BiGRU-Based Bimodal Feature Fusion Model for English Listening and Speaking Recognition
Gao, Y., & Wang, W. (2026) · International Conference on Machine Intelligence and Digital Applications (MIDA) · doi:10.1145/3801438.3803158
Ai-Powered Visual Communication Assistant for Deaf and Dumb
Madhu, S., Ali, B., Sarma, D. P. M. M. S., Sravanthi, M., Prasad, R. D., & Showkat, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27448-9_14
Thi, D. N., Vu, D.-Q., Bao, T. M., & Le, D. (2026) · International Journal of Artificial Intelligence and Machine Learning · doi:10.51483/ijaiml.6.12s.2026.806-825
Dobre, A. L., Țapu, R., SULTANA, A.-E., & Mocanu, B. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135349
Fatta, H. A., Pristyanto, Y., Kuswanto, J., Aziza, R. F. A., & Ramadhan, A. (2026) · International journal of intelligent engineering and systems · doi:10.22266/ijies2026.1130.47
Metadata
- Authors
- Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L.
- Affiliations
- National University of Defense Technology (CN)
- University of Oulu (FI)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-09
- Volume
- 46
- Issue
- 9
- Pages
- 6001-6022
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4392780157
- Cited by
- 64
- References
- 187
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 18
- FWCI
- 5.05
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2021YFB3100800)
- Academy of Finland (331883)
- National Natural Science Foundation of China (62376283, 61872379)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- visual speech, visual speech recognition, deep learning, benchmark dataset, research challenges
- MeSH terms
- Deep Learning, Humans, Image Processing, Computer-Assisted, Speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.