Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 201–300 of 1,001
Audio-Driven Facial Animation with Deep Learning: A Survey
Jiang, D., Chang, J. M., You, L., Bian, S., Kosk, R., & Maguire, G. (2024) · Information · doi:10.3390/info15110675 · Cited by 38
FT2TF: First-Person Statement Text-to-Talking Face Generation
Diao, X., Cheng, M., Barrios, W., & Jin, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00472 · Cited by 38
UltraSpeech: Speech Enhancement by Interaction between Ultrasound and Speech
Ding, H., Wang, Y., Li, H., Zhao, C., Wang, G., Wei, X., & Zhao, J. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3550303 · Cited by 38
Visual Speech Recognition in a Driver Assistance System
Ivanko, D., Ryumin, D., Kashevnik, A., Axyonov, A., & Karnov, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909819 · Cited by 38
Speaker-Adaptive Lip Reading with User-Dependent Padding
Kim, M., Kim, H., & Ro, Y. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_33 · Cited by 38
Yeo, J. H., Kim, M., Choi, J., Kim, D. H., & Ro, Y. M. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3352388 · Cited by 38
Cheng, X., Jin, T., Huang, R., Li, L., Wang, L., Wang, Z., Wang, Y., Liu, H., Yin, A., & Zhao, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01442 · Cited by 38
Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
Jang, Y., Kim, J., Ahn, J., Kwak, D., Yang, H.-S., Ju, Y.-C., Kim, I.-H., Kim, B.-Y., & Chung, J. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00842 · Cited by 37
Self-citationWang, Z., Wu, S., Chen, H., He, M., Du, J., Lee, C., Chen, J., Watanabe, S., Siniscalchi, S. M., Scharenborg, O., et al. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094836 · Cited by 37
Target Active Speaker Detection with Audio-visual Cues
Jiang, Y., Tao, R., Pan, Z., & Li, H. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-574 · Cited by 37
He, Y., Seng, K. P., & Ang, L.-M. (2023) · Information · doi:10.3390/info14100575 · Cited by 37
Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning
Li, W., Zhao, X.-L., Ma, Z., Wang, X., Fan, X., & Tian, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611759 · Cited by 37
Boosting Multimodal Learning via Disentangled Gradient Learning
Wei, S., Luo, C., & Luo, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02124 · Cited by 36
Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations
Oneaţă, D., & Cucu, H. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00504 · Cited by 36
Developing phoneme‐based lip‐reading sentences system for silent speech recognition
El‐Bialy, R., Chen, D., Fenghour, S., Hussein, W., Xiao, P., Karam, O. H., & Li, B. (2022) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12131 · Cited by 37
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
Seo, P. H., Nagrani, A., & Schmid, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02195 · Cited by 36
Visualtts: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over
Lu, J., Şişman, B., Liu, R., Zhang, M., & Li, H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746421 · Cited by 35
Cross-modal text and visual generation: A systematic review. Part 1: Image to text
Żelaszczyk, M., & Mańdziuk, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.008 · Cited by 35
Yeo, J. H., Kim, M., Watanabe, S., & Ro, Y. M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446720 · Cited by 35
XVO: Generalized Visual Odometry via Cross-Modal Self-Training
Lai, L., Shangguan, Z., Zhang, J., & Ohn-Bar, E. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00926 · Cited by 35
Zhang, Y., Li, Z., Wang, D., Zhang, J., Zhou, D., Yin, Z., Dai, X., Yu, G., & Li, X. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2507.09862 · Cited by 35
When deep learning meets watermarking: A survey of application, attacks and defenses
Chen, H., Liu, C., Zhu, T., & Zhou, W. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103830 · Cited by 34
Audio-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis
Chen, H., Du, J., Dai, Y., Lee, C., Siniscalchi, S. M., Watanabe, S., Scharenborg, O., Chen, J., Yin, B., & Pan, J. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10483 · Cited by 33
Advances and Challenges in Deep Lip ReadingPreprint
Oghbaie, M., Sabaghi, A., Hashemifard, K., & Akbari, M. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07879 · Cited by 33
Heidarivincheh, F., McConville, R., Morgan, C., McNaney, R., Masullo, A., Mirmehdi, M., Whone, A. L., & Craddock, I. J. (2021) · Sensors · doi:10.3390/s21124133 · Cited by 32
Speech Reconstruction With Reminiscent Sound Via Visual Voice Memory
Hong, J. H., Kim, M., Park, S. J., & Ro, Y. M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3126925 · Cited by 32
Multi-Modal Acoustic-Articulatory Feature Fusion For Dysarthric Speech Recognition
Yue, Z., Loweimi, E., Cvetković, Z., Christensen, H., & Barker, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746855 · Cited by 32
Audio-Visual End-to-End Multi-Channel Speech Separation, Dereverberation and Recognition
Li, G., Deng, J., Geng, M., Jin, Z., Wang, T., Hu, S., Cui, M., Meng, H. M. L., & Liu, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3294705 · Cited by 32
A multimodal dynamical variational autoencoder for audiovisual speech representation learning
Sadok, S., Leglaive, S., Girin, L., Alameda-Pineda, X., & Séguier, R. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106120 · Cited by 32
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
Lai, Y.-H., Chen, Y.-C., & Wang, F. (2023) · neural information processing systems · doi:10.52202/075280-3221 · Cited by 32
TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning
Jung, C., Lee, S., Nam, K.-H., Rho, K., Kim, Y. J., Jang, Y., & Chung, J. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448124 · Cited by 31
Self-citationEgo4D: Around the World in 3,600 Hours of Egocentric Video
Grauman, K. L., Westbury, A., Byrne, E., Cartillier, V., Chavis, Z. Q., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Kukreja, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3381075 · Cited by 31
Adversarial Attacks to Multi-Modal Models
Dou, Z., Hu, X., Yang, H., Liu, Z., & Fang, M. (2023) · doi:10.1145/3689217.3690619 · Cited by 31
Audio-Visual Target Speaker Extraction With Selective Auditory Attention
Tao, R., Qian, X., Jiang, Y., Li, J., Wang, J., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3527766 · Cited by 31
Development of Output Correction Methodology for Long Short Term Memory-Based Speech Recognition
Arslan, R. S., & Barışçı, N. (2019) · Sustainability · doi:10.3390/su11154250 · Cited by 31
Hu, Y., Li, R., Chen, C., Qin, C., Zhu, Q., & Chng, E. S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.848 · Cited by 31
Hu, Y., Chen, C., Li, R., Zou, H., & Chng, E. S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.649 · Cited by 31
Gul, S., & Khan, M. S. (2023) · IEEE Access · doi:10.1109/access.2023.3344813 · Cited by 31
Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition
Wang, J., Pan, Z., Zhang, M., Tan, R. T., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29882 · Cited by 31
Ahn, Y. J., Park, J., Park, S., Choi, J., & Kim, K.-E. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-432 · Cited by 31
Koutlis, C., & Papadopoulos, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.10193 · Cited by 31
Lohrenz, T., Li, Z., & Fingscheidt, T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-555 · Cited by 30
Audio-Visual Multi-Channel Integration and Recognition of Overlapped Speech
Yu, J., Zhang, S.-X., Wu, B., Liu, S., Hu, S., Geng, M., Liu, X., Meng, H. M. L., & Yu, D. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3078883 · Cited by 30
Review on research progress of machine lip reading
Pu, G., & Wang, H. (2022) · The Visual Computer · doi:10.1007/s00371-022-02511-4 · Cited by 30
Learning Visual Styles from Audio-Visual Associations
Li, T., Liu, Y., Owens, A., & Zhao, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_14 · Cited by 30
Information cascades prediction with attention neural network
Liu, Y., Bao, Z., Zhang, Z., Tang, D., & Xiong, F. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-020-00218-w · Cited by 29
Bimodal variational autoencoder for audiovisual speech recognition
Sayed, H. M., ElDeeb, H. E., & Taie, S. A. (2021) · Machine Learning · doi:10.1007/s10994-021-06112-5 · Cited by 29
Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model
Martel, H., Richter, J., Li, K., Hu, X., & Gerkmann, T. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1753 · Cited by 29
Emotionally Enhanced Talking Face Generation
Goyal, S., Bhagat, S., Uppal, S., Jangra, H., Yu, Y., Yin, Y., & Shah, R. R. (2023) · International Workshop on Multimedia Content Generation and Evaluation: New Methods and Practice (McGE) · doi:10.1145/3607541.3616812 · Cited by 29
IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech SeparationPreprint
Li, K., Yang, R., Sun, F., & Hu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.08143 · Cited by 29
LaughTalk: Expressive 3D Talking Head Generation with Laughter
Kim, S.-B., Hyun, L., Hong, D. H., Nam, S., Ju, J., & Oh, T.-H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00628 · Cited by 28
Dual-CNN based multi-modal sleep scoring with temporal correlation driven fine-tuning
Zhang, L., Chen, D., Chen, P., Li, W., & Li, X. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.08.020 · Cited by 28
Audio–Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
Lee, Y.-H., Jang, D., Kim, J.-B., Park, R., & Park, H. (2020) · Applied Sciences · doi:10.3390/app10207263 · Cited by 28
Shang, L., Zhang, Y., Deng, Y., & Wang, D. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3533919 · Cited by 27
Electromyogram‐Based Lip‐Reading via Unobtrusive Dry Electrodes and Machine Learning Methods
Dong, P., Song, Y., Yu, S., Zhang, Z., Mallipattu, S. K., Djurić, P. M., & Yao, S. (2023) · Small · doi:10.1002/smll.202205058 · Cited by 27
Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic Sensing
Zhang, Q., Lan, Y., Guo, K., & Wang, D. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659614 · Cited by 27
RUSAVIC Corpus: Russian Audio-Visual Speech in Cars
Ivanko, D., Axyonov, A., Ryumin, D., Kashevnik, A. M., & Karpov, A. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/46wjh8y2d29k · Cited by 27
LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC modelsPreprint
Margam, D. K., Aralikatti, R. C., Sharma, T., Thanda, A., Pujitha, A. K., Roy, S., & Venkatesan, S. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.12170 · Cited by 27
Advances in Speech Separation: Techniques, Challenges, and Future TrendsPreprint
Li, K., Chen, G., Sang, W., Luo, Y., Chen, Z., Wang, S., He, S., Wang, Z.-Q., Li, A., Wu, Z., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.10830 · Cited by 27
SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer
Park, Y.-H., Park, R., & Park, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130289 · Cited by 26
DMMAN: A two-stage audio–visual fusion framework for sound separation and event localization
Hu, R., Zhou, S., Tang, Z. R., Chang, S., Huang, Q., Liu, Y., Han, W., & Wu, E. Q. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.10.003 · Cited by 26
Kumar, T., Mahrishi, M., & Meena, G. (2021) · Learning and analytics in intelligent systems · doi:10.1007/978-3-030-85383-9_8 · Cited by 26
Rathipriya, N., & Maheswari, N. (2024) · IEEE Access · doi:10.1109/access.2024.3463969 · Cited by 26
Choi, J., Park, S. J., Kim, M., & Ro, Y. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02580 · Cited by 26
Audio-visual representation learning via knowledge distillation from speech foundation models
Zhang, J.-X., Wan, G., Gao, J., & Ling, Z.-H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111432 · Cited by 26
Classification of Sonar Targets in Air: A Neural Network Approach
Kroh, P. K., Simon, R., & Rupitsch, S. J. (2019) · Sensors · doi:10.3390/s19051176 · Cited by 26
Multi-Modal Embeddings Using Multi-Task Learning for Emotion Recognition
Khare, A., Parthasarathy, S., & Sundaram, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1827 · Cited by 26
From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from SpeechPreprint
Choi, H.-S., Park, C., & Lee, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05830 · Cited by 26
Yeo, J. H., Kim, M., Kim, C. W., Petridis, S., & Ro, Y. M. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00630 · Cited by 25
Contribution-Aware Dynamic Multi-Modal Balance for Audio-Visual Speech Separation
Xu, X., Tu, W., Yang, Y., Li, J., Yang, Y., & Chen, H. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3654399 · Cited by 25
VALLR: Visual ASR Language Model for Lip Reading
Thomas, M., Fish, E., & Bowden, R. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00273 · Cited by 25
AVMSN: An Audio-Visual Two Stream Crowd Counting Framework Under Low-Quality Conditions
Hu, R., Mo, Q., Xie, Y., Xu, Y., Chen, J., Yang, Y., Zhou, H., Tang, Z., & Wu, E. Q. (2021) · IEEE Access · doi:10.1109/access.2021.3074797 · Cited by 25
Zhai, B., Guan, Y., Catt, M., & Plötz, T. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3494961 · Cited by 25
Hierarchical multi-label taxonomic classification of carbonate skeletal grains with deep learning
Ho, M., Idgunji, S., Payne, J. L., & Koeshidayatullah, A. (2022) · Sedimentary Geology · doi:10.1016/j.sedgeo.2022.106298 · Cited by 25
Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos
Waibel, A. H., Behr, M., Yaman, D., Eyiokur, F. İ., Nguyen, T. N., Mullov, C., Demirtaş, M. A., Kantarcı, A., Constantin, S., & Ekenel, H. K. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193719 · Cited by 25
TellMeTalk: Multimodal-driven talking face video generation
Li, P., Zhao, H., Liu, Q., Tang, P., & Zhang, L. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.109049 · Cited by 25
Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation
Yaman, D., Eyiokur, F. İ., Bärmann, L., Aktı, Ş., Ekenel, H. K., & Waibel, A. H. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00607 · Cited by 25
Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception
Wang, J., Qian, X., & Li, H. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3507575 · Cited by 25
TieLent: A Casual Neck-Mounted Mouth Capturing Device for Silent Speech Interaction
Kimura, N., Hayashi, K., & Rekimoto, J. (2020) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3399715.3399852 · Cited by 25
An Implementation of Multimodal Fusion System for Intelligent Digital Human GenerationPreprint
Zhou, Y., Chen, Y., Bi, K., Xiong, L., & Liu, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.20251 · Cited by 25
Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild
He, Y., Seng, K. P., & Ang, L.-M. (2023) · Sensors · doi:10.3390/s23041834 · Cited by 24
Stochastic Latent Talking Face Generation Toward Emotional Expressions and Head Poses
Sheng, Z., Nie, L., Zhang, M., Chang, X., & Yan, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3311039 · Cited by 24
Audiovisual speech recognition based on a deep convolutional neural network
Shashidhar, R., Patilkulkarni, S., Ravi, V., Gururaj, H. L., & Krichen, M. (2023) · Data Science and Management · doi:10.1016/j.dsm.2023.10.002 · Cited by 24
Djilali, Y. A. D., Narayan, S., Boussaid, H., Almazrouei, E., & Debbah, M. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01268 · Cited by 24
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer
Burchi, M., Puvvada, K. C., Balam, J., Ginsburg, B., & Timofte, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445891 · Cited by 24
Han, H., Anwar, M. A., Pino, J., Hsu, W.-N., Carpuat, M. J., Shi, B., & Wang, C. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.697 · Cited by 24
Kim, S., Jang, K., Bae, S., Kim, H., & Yun, S.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832305 · Cited by 24
Audio-Visual Multi-Channel Recognition of Overlapped Speech
Yu, J., Wu, B., Gu, R., Zhang, S.-X., Chen, L., Xu, Y., Meng, Y., Su, D., Yu, D., Liu, X., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2346 · Cited by 24
Influence of Autoencoder-Based Data Augmentation on Deep Learning-Based Wireless Communication
Li, L., Zhang, Z. J., & Yang, L. (2021) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2021.3092716 · Cited by 24
Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech RecognitionPreprint
Wei, L., Zhang, J., Hou, J., & Dai, L.-R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02686 · Cited by 24
Kalkhorani, V. A., Yu, C., Kumar, A., Tan, K., Xu, B., & Wang, D. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3567838 · Cited by 23
Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild
Hegde, S. B., Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548081 · Cited by 23
Self-Supervised Audio-Visual Speech Representations Learning by Multimodal Self-Distillation
Zhang, J., Wan, G., Ling, Z.-H., Pan, J., Gao, J., & Liu, C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095029 · Cited by 23
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Cheng, X., Jin, T., Li, L., Wang, L., Duan, X., & Zhao, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.363 · Cited by 23
Expressive Speech-Driven Facial Animation with Controllable Emotions
Chen, Y., Zhao, J., & Zhang, W.-Q. (2023) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew59549.2023.00073 · Cited by 23
Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video
Wu, X., Hu, P., Wu, Y., Lyu, X., Cao, Y., Shan, Y., Yang, W., Sun, Z., & Qi, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02026 · Cited by 23
Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction
Mu, Z., & Yang, X. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/709 · Cited by 23
Boosting Speech Recognition Robustness to Modality-Distortion with Contrast-Augmented Prompts
Fu, D., Cheng, X., Yang, X., Hanting, W., Zhao, Z., & Jin, T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681347 · Cited by 23
MMAL: Multi-Modal Analytic Learning for Exemplar-Free Audio-Visual Class Incremental Tasks
Yue, X., Zhang, X., Chen, Y., Zhang, C., Lao, M., Zhuang, H., Qian, X., & Li, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681607 · Cited by 23
Audio Adversarial Examples Generation with Recurrent Neural Networks
Chang, K.-H., Huang, P.-H., Yu, H., Jin, Y., & Wang, T.-C. (2020) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac47756.2020.9045597 · Cited by 23
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.