Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 187 of the 188 references it cites were found in a registry or an index, and 1 are not checked yet; 10 of the 273 sources its claims cite were checked for support.

Deep Learning for Visual Speech Analysis: A Survey

Open access
Sheng, Changchong, Kuang, Gangyao, Bai, Liang, Hou, Chenping, Guo, Yulan, Xu, Xin, Pietikäinen, Matti, Liu, Li·2024·IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

Verified 2026-10-11 · 1 reference not checked yet

Overview

CoChat Summary

## TL;DR

This survey reviews deep learning methods for visual speech analysis, focusing on the two dual tasks of visual speech recognition (VSR) and visual speech generation (VSG). It summarizes datasets, challenges, evaluation metrics, representative methods, and reported state-of-the-art results, and it highlights open problems and future directions.

## Key findings

  • The paper frames VSA around two formally dual problems: VSR and VSG, and argues that VSR and VSG share core technical points such as visual speech representation learning and contextual sequence modeling.
  • For VSR, the paper reports strong recent performance: the SOTA on LRW reached 88.5% and on LRW-1000 50.5% with cross-modal mutual learning; on the simple GRID dataset, deep VSR models almost saturate at 1.3% WER.
  • For word-level VSR, a baseline C3D-ResNet34 + BiLSTM achieved 83.5% on LRW, and later methods improved to 87.7% on LRW; MS-TCN-based methods are reported at 85.3% on LRW and 41.4% on LRW-1000.
  • For VSG, the paper compares representative methods using PSNR, SSIM, and LMD, and reports examples such as 29.36 PSNR / 0.74 SSIM / 1.35 LMD for a one-stage method on VoxCeleb+LRW, and 33.4 PSNR / 0.96 SSIM / 0.60 LMD for Prajwal et al. on LRS2.

## Methods

  • Survey design: a comprehensive literature review organized around two main tasks, VSR and VSG, plus datasets, evaluation metrics, and practical challenges.
  • Coverage: discusses benchmark datasets in controlled and uncontrolled settings, with examples including AVICAR, GRID, OuluVS2, LRW (> 1,000k word instances), LRS2-BBC (> 144.5k utterances), VoxCeleb1 (> 100k utterances from 1,251 celebrities), LRS3-TED (> 400 hours), and MuAViC (1,200 hours in 9 languages).
  • Analysis: synthesizes methods by taxonomy and reports representative performance comparisons from published results rather than conducting new experiments.

## Limitations

  • The paper explicitly says that many practical issues remain unresolved, including the need for fewer labels, better benchmark fairness, improved robustness analysis, and better evaluation benchmarks for VSG.
  • It states that fair comparison is difficult because VSR/VSG results are affected by extra language models, extra training datasets, audio signals, implementation details, and non-unified test platforms.
  • Evident from the text: this is a survey, so it does not present a new experimental dataset, model, or unified benchmark of its own; instead it depends on heterogeneous reported results from prior work.

Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3376710

Cites

187 of 188 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

9 Valid · 50 Not valid · 176 Not checked · 132 Own results and interpretations

97 more claims · 1 with their cited sources checked

Create a free account to see all 367 claims and to verify the 263 references still open.

See all claims — free account
  • Probably not a substantial claim. 216
  • It may state two separate claims as one. 106

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 187 references

Research works printed but not linked: 6

  1. He, Dual learning for machine translation, 820
  2. Goodfellow, Generative adversarial nets, 2672
  3. Weng, Learning spatio temporal features with two stream deep 3D CNNs for lipreading
  4. Chung, You said that?
  5. 10.1609/aaai.v36i3.20154 · DOI
  6. Ye, GeneFace: Generalized and high-fidelity audio-driven 3D talking face synthesis

Cited by

See all 73

Metadata

Authors
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L.
Affiliations
  • National University of Defense Technology (CN)
  • University of Oulu (FI)
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2024-09
Volume
46
Issue
9
Pages
6001-6022
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4392780157
Cited by
64
References
187
Citations by year
2026: 18 · 2025: 23 · 2024: 18
FWCI
5.05
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Key Research and Development Program of China (2021YFB3100800)
  • Academy of Finland (331883)
  • National Natural Science Foundation of China (62376283, 61872379)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
visual speech, visual speech recognition, deep learning, benchmark dataset, research challenges
MeSH terms
Deep Learning, Humans, Image Processing, Computer-Assisted, Speech

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep Learning for Visual Speech Analysis: A Survey. https://app.cochat.ai/papers/10.1109/tpami.2024.3376710
The paper
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024). Deep Learning for Visual Speech Analysis: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3376710

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.