The DOI matches its Crossref record; 187 of the 188 references it cites were found in a registry or an index, and 1 are not checked yet; 10 of the 273 sources its claims cite were checked for support.
Deep Learning for Visual Speech Analysis: A Survey
Open accessAbstract
Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.
Verified 2026-10-11 · 1 reference not checked yet
Overview
CoChat Summary
## TL;DR
This survey reviews deep learning methods for visual speech analysis, focusing on the two dual tasks of visual speech recognition (VSR) and visual speech generation (VSG). It summarizes datasets, challenges, evaluation metrics, representative methods, and reported state-of-the-art results, and it highlights open problems and future directions.
## Key findings
- The paper frames VSA around two formally dual problems: VSR and VSG, and argues that VSR and VSG share core technical points such as visual speech representation learning and contextual sequence modeling.
- For VSR, the paper reports strong recent performance: the SOTA on LRW reached 88.5% and on LRW-1000 50.5% with cross-modal mutual learning; on the simple GRID dataset, deep VSR models almost saturate at 1.3% WER.
- For word-level VSR, a baseline C3D-ResNet34 + BiLSTM achieved 83.5% on LRW, and later methods improved to 87.7% on LRW; MS-TCN-based methods are reported at 85.3% on LRW and 41.4% on LRW-1000.
- For VSG, the paper compares representative methods using PSNR, SSIM, and LMD, and reports examples such as 29.36 PSNR / 0.74 SSIM / 1.35 LMD for a one-stage method on VoxCeleb+LRW, and 33.4 PSNR / 0.96 SSIM / 0.60 LMD for Prajwal et al. on LRS2.
## Methods
- Survey design: a comprehensive literature review organized around two main tasks, VSR and VSG, plus datasets, evaluation metrics, and practical challenges.
- Coverage: discusses benchmark datasets in controlled and uncontrolled settings, with examples including AVICAR, GRID, OuluVS2, LRW (> 1,000k word instances), LRS2-BBC (> 144.5k utterances), VoxCeleb1 (> 100k utterances from 1,251 celebrities), LRS3-TED (> 400 hours), and MuAViC (1,200 hours in 9 languages).
- Analysis: synthesizes methods by taxonomy and reports representative performance comparisons from published results rather than conducting new experiments.
## Limitations
- The paper explicitly says that many practical issues remain unresolved, including the need for fewer labels, better benchmark fairness, improved robustness analysis, and better evaluation benchmarks for VSG.
- It states that fair comparison is difficult because VSR/VSG results are affected by extra language models, extra training datasets, audio signals, implementation details, and non-unified test platforms.
- Evident from the text: this is a survey, so it does not present a new experimental dataset, model, or unified benchmark of its own; instead it depends on heterogeneous reported results from prior work.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 50 Not valid · 176 Not checked · 132 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Meanwhile, the emergence of large-scale VSA datasets [28, 29, 30, 31, 32] promoted the further development of deep learning based VSA research.”
What the sources say
- Not checked yetcites [28] source
- Not checked yetcites [29] source
- Supportedcites [30] source
The Introduction (p. 1) presents VoxCeleb2 as a large-scale speaker-recognition dataset addressing a shortage of such data. The paper also reports training a deep CNN speaker-recognition system on it and achieving state-of-the-art verification performance (p. 1; Conclusion, p. 4). This supports its use as an example of a large-scale dataset advancing deep-learning-based speaker recognition. (LLM, conf=0.90)
- Supportedcites [31] source
The Abstract identifies VoxCeleb as a large-scale speaker dataset and reports CNN-based identification and verification results on it. Related Works says a large real-world dataset is essential to the shift toward deep CNNs for speaker identification, supporting the cited connection between such datasets and deep-learning research. (LLM, conf=0.91)
- Not checked yetcites [32] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Many deep learning driven technologies [39, 105, 106, 107] have been introduced to address the alignment of AV signals in a self-supervised way, i . e ., automatically creating positive and negative A-V sync/out-of-sync pairs.”
What the sources say
- Supportedcites [39] source
Section 3.3.1 describes an audio-video sync discriminator trained with automatically sampled in-sync and out-of-sync pairs: an audio window is taken either from the matching time or a different time. Section 3.3.2 says the paper adapts and trains that discriminator for its own lip-generation method, supporting its use as an example of the technology described. (LLM, conf=0.90)
- Not checked yetcites [105] source
- Not checked yetcites [106] source
- Not checked yetcites [107] source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“[11] is the first work introducing transformers to VSR.”
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 3 (page unavailable) describes Transformer-based audio-visual speech-recognition models, supporting the claim that this work uses Transformers for VSR. But the Introduction (page unavailable) says one Transformer variant had already been published in [2]; our copy does not establish that this paper was the first to introduce Transformers to VSR. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
The Introduction and §6.1 (Lips only; page unavailable) report that the Transformer-based model improved sentence-level visual speech recognition from 70.4% to 48.3% word error rate relative to the earlier model. But the Introduction says the visual frontend was also updated and attributes the improvement to both frontend and backend changes; the experiments do not isolate a Transformer-versus-RNN backend effect. (LLM, conf=0.87)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Except for marginbased loss, L1 loss and binary classification loss [48, 149, 150, 151] are also widely used for A-V representations learning.”
What the sources say
- Not checked yetcites [48] source
- Partly supportedcites [149] sourceExperimental — accuracy not yet measured
Our copy of Senocak et al. (CVPR 2018) supports only the binary-classification half of the claim: in Sec. 4 (Eq. 7) the supervised loss L_S is a cross-entropy over the attention map whose ground-truth values α_GT,i are binary, and the paper notes it performs slightly better than the mean-square-error alternative. The L1-loss part is not in our copy — the unsupervised objective is the distance-ratio/triplet loss (Eq. 5, Sec. 4) built on ℓ2 distances, and no ℓ1 loss appears anywhere (searched: L1 loss, distance/embedding objective, loss ablation). The paper also makes no statement that either loss is 'widely used' for audio-visual representation learning; it is a single method paper, so the survey-level framing rests on the other citations in the cluster. (LLM, conf=0.68)
- Not checked yetcites [150] source
- Not checked yetcites [151] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
97 more claims · 1 with their cited sources checked
Create a free account to see all 367 claims and to verify the 263 references still open.
- Probably not a substantial claim. 216
- It may state two separate claims as one. 106
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1.2 Datasets under uncontrolled environments
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
§ 3.1.2 Datasets under uncontrolled environments
“Compared to LRS2-TED, it has a larger scale in terms of duration, vocabulary, and number of speakers.”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Research works printed but not linked: 6
- He, Dual learning for machine translation, 820
- Goodfellow, Generative adversarial nets, 2672
- Weng, Learning spatio temporal features with two stream deep 3D CNNs for lipreading
- Chung, You said that?
- 10.1609/aaai.v36i3.20154 · DOI
- Ye, GeneFace: Generalized and high-fidelity audio-driven 3D talking face synthesis
Cited by
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Peng, Z., Wu, H., Song, Z., Xu, H., Zhu, X., He, J., Liu, H., & Fan, Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01891 · Cited by 159
Xing, Z., Ma, G., Wang, L., Yang, L., Guo, X., & Chen, S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3546874 · Cited by 98
Talking human face generation: A survey
Toshpulatov, M. A., Lee, W., & Lee, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119678 · Cited by 43
CNN Learning Strategy for Recognizing Facial Expressions
Lee, D., & Yoo, J. (2023) · IEEE Access · doi:10.1109/access.2023.3294099 · Cited by 37
A Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 32
Metadata
- Authors
- Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L.
- Affiliations
- National University of Defense Technology (CN)
- University of Oulu (FI)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-09
- Volume
- 46
- Issue
- 9
- Pages
- 6001-6022
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4392780157
- Cited by
- 64
- References
- 187
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 18
- FWCI
- 5.05
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2021YFB3100800)
- Academy of Finland (331883)
- National Natural Science Foundation of China (62376283, 61872379)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- visual speech, visual speech recognition, deep learning, benchmark dataset, research challenges
- MeSH terms
- Deep Learning, Humans, Image Processing, Computer-Assisted, Speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Learning for Visual Speech Analysis: A Survey. https://app.cochat.ai/papers/10.1109/tpami.2024.3376710
- The paper
- Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024). Deep Learning for Visual Speech Analysis: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3376710
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.