Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 401–500 of 1,001
Robust Audio-Visual Speech Recognition Based on Hybrid Fusion
Liu, H., Li, W., & Yang, B. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412817 · Cited by 15
Watch to Listen Clearly: Visual Speech Enhancement Driven Multi-modality Speech Recognition
Xu, B., Wang, J., Lu, C., & Guo, Y. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093314 · Cited by 15
Learning De-identified Representations of Prosody from Raw AudioPreprint
Weston, J., Lenain, R., Meepegama, U., & Fristed, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08248 · Cited by 15
Multi-task Learning Approach for Automatic Modulation and Wireless Signal ClassificationPreprint
Jagannath, A., & Jagannath, J. (2021) · doi:10.36227/techrxiv.15156978 · Cited by 15
Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth InconsistenciesPreprint
Datta, S. K., Jia, S., & Lyu, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.01470 · Cited by 15
Learning Speaker-specific Lip-to-Speech Generation
Varshney, M., Yadav, R., Namboodiri, V. P., & Hegde, R. M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956600 · Cited by 14
Bommareddy, S., Samyal, T., & Dahiya, S. (2023) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit59222.2023.10205614 · Cited by 14
WASD: A Wilder Active Speaker Detection Dataset
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3412821 · Cited by 14
Drug repositioning for Parkinson’s disease: An emphasis on artificial intelligence approaches
Karimi-Sani, I., Sharifi, M., Abolpour, N., Lotfi, M., Atapour, A., Takhshid, M. A., & Sahebkar, A. (2025) · Ageing Research Reviews · doi:10.1016/j.arr.2024.102651 · Cited by 14
AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
Choi, J., Kim, J.-H., Sung-Bin, K., Oh, T.-H., & Chung, J. S. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754577 · Cited by 14
Self-citationPeng, Z., Hu, W., Ma, J., Zhu, X., Zhang, X. M., Zhao, H., Tian, H., He, J., Liu, H., & Fan, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3630057 · Cited by 14
Speaker-Independent Lipreading By Disentangled Representation Learning
Zhang, Q., Wang, S., & Chen, G. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506396 · Cited by 14
Zhang, Y., He, W., Li, M., Tian, K., Zhang, Z., Cheng, J., Wang, Y., & Liao, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747284 · Cited by 14
Braga, O., & Siohan, O. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746036 · Cited by 14
Zheng, Q., Hong, J., & Farazi, M. R. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191705 · Cited by 14
Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-Talker Speech
Li, J., Tao, R., Pan, Z., Ge, M., Wang, S., & Li, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448398 · Cited by 14
HyperLips: hyper control lips with high resolution decoder for talking face generation
Chen, Y., Yao, Y., Li, Z., Wang, W., Zhang, Y., Yang, H. S., & Wen, X. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05914-z · Cited by 14
Speaker-Independent Visual Speech Recognition with the Inception V3 Model
Santos, T. I., Abel, A. K., Wilson, N., & Xu, Y. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383540 · Cited by 14
End-To-End Lip Synchronisation Based on Pattern Classification
Kim, Y. J., Heo, H. S., Chung, S.-W., & Lee, B. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383616 · Cited by 14
Convolutional Neural Networks for Speech Controlled Prosthetic Hands
Jafarzadeh, M., & Tadesse, Y. T. (2019) · International Conference on Transdisciplinary AI (TransAI) · doi:10.1109/transai46475.2019.00014 · Cited by 14
Cross-Modal Language Modeling in Multi-Dynamic-Informed Context for Lip Reading
Ai, X., & Fang, B. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3282109 · Cited by 13
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
Kurzendörfer, D., Mercea, O.-B., Koepke, A. S., & Akata, Z. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00269 · Cited by 13
VPT: Video portraits transformer for realistic talking face generation
Zhang, Z., Zhang, J., & Mai, W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107122 · Cited by 13
Mixed Precision DNN Quantization for Overlapped Speech Separation and Recognition
Xu, J., Yu, J., Liu, X., & Meng, H. M. L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746885 · Cited by 13
Transformer-Based Lip-Reading with Regularized Dropout and Relaxed Attention
Li, Z., Lohrenz, T., Dunkelberg, M., & Fingscheidt, T. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023442 · Cited by 13
Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech
Yu, C., Yu, J., Qian, Z., & Tan, Y. (2023) · Sensors · doi:10.3390/s23042071 · Cited by 13
Dai, Y., Chen, H., Du, J., Ding, X., Ding, N., Jiang, F., & Lee, C. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00447 · Cited by 13
Exploring Phonetic Context-Aware Lip-Sync for Talking Face Generation
Park, S. J., Kim, M., Choi, J., & Ro, Y. M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447284 · Cited by 13
ViLaS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition
Ni, Z., Han, M., Chen, F., Meng, L., Shi, J. Y., Lv, P., & Xu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448450 · Cited by 13
EDSep: An Effective Diffusion-Based Method for Speech Source Separation
Dong, J., Wang, X., & Mao, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889348 · Cited by 13
Speaker-Dependent Visual Command Recognition in Vehicle Cabin: Methodology and Evaluation
Ivanko, D., Ryumin, D., Axyonov, A., & Kashevnik, A. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87802-3_27 · Cited by 13
Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition
Li, G., Yu, J., Deng, J., Liu, X., & Meng, H. M. L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747237 · Cited by 13
Visual Keyword Spotting with AttentionPreprint
Prajwal, K. R., Momeni, L., Afouras, T., & Zisserman, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15957 · Cited by 13
Self-citationLook, Listen and Recognise: Character-Aware Audio-Visual Subtitling
Korbar, B., Huh, J., & Zisserman, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446480 · Cited by 12
Self-citationMulti-modal co-learning for silent speech recognition based on ultrasound tongue images
Guo, M., Wei, J., Zhang, R., Zhao, Y., & Fang, Q. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103140 · Cited by 12
Leveraging AI for Advancements in Qualitative Research Methodology
Haouam, I. (2025) · Journal on artificial intelligence · doi:10.32604/jai.2025.064145 · Cited by 12
VGGSounder: Audio-Visual Evaluations for Foundation Models
Zverev, D., Wiedemer, T., Prabhu, A., Bethge, M., Brendel, W., & Koepke, A. S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00103 · Cited by 12
Rahul, C., Arathi, T., Panicker, L. S., & Gopikakumari, R. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105051 · Cited by 12
Slow-Fast Time Parameter Aggregation Network for Class-Incremental Lip Reading
Zhang, X., Zhang, C., Wang, T., Tang, J., Lao, S., & Li, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612460 · Cited by 12
Audio-visual saliency prediction with multisensory perception and integration
Xie, J., Liu, Z., Li, G., & Song, Y. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104955 · Cited by 12
OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset
Park, J., Hwang, J.-W., Choi, K., Lee, S.-H., Ahn, J. H., Park, R., & Park, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446901 · Cited by 12
Cross-Attention Fusion of Visual and Geometric Features for Large-Vocabulary Arabic Lipreading
Daou, S., Hamadou, A. B., Rekik, A., & Kallel, A. (2025) · Technologies · doi:10.3390/technologies13010026 · Cited by 12
A Survey of Talking Head Synthesis Techniques: Portrait Generation, Driving Mechanisms, and Editing
Meng, M., Zhao, Y., Zhang, B., Zhu, Y., Shi, W., Wen, Z., & Fan, Z. (2026) · ACM Computing Surveys · doi:10.1145/3785656 · Cited by 13
Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning
Yang, J. C., Li, Z., Xie, S., Yu, W., Li, S., & Du, B. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/43f8gdht9kqm · Cited by 12
Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization
Wu, L., Zhang, X., Zhang, Y., Zheng, C., Liu, T., Xie, L., Yan, Y., & Yin, E. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/28em5tq9jc2n · Cited by 12
Sterpu, G., Saam, C., & Harte, N. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2480 · Cited by 12
Multichannel speech separation using hybrid GOMF and enthalpy-based deep neural networks
Koteswararao, Y. V., & Rao, C. B. R. (2021) · Multimedia Systems · doi:10.1007/s00530-020-00740-y · Cited by 12
Development of Visual and Audio Speech Recognition Systems Using Deep Neural Networks
Ivanko, D., & Ryumin, D. (2021) · International Conference on Computer Graphics and Vision (GraphiCon) · doi:10.20948/graphicon-2021-3027-905-916 · Cited by 12
Is Lip Region-of-Interest Sufficient for Lipreading?
Zhang, J.-X., Wan, G., & Pan, J. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556571 · Cited by 12
Visual Keyword Spotting with Attention
Prajwal, K. R., Momeni, L., Afouras, T., & Zisserman, A. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.212 · Cited by 12
Self-citationLIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2v2smyvuxcgv · Cited by 12
Performance Analysis of Isolated Speech Recognition Technique Using MFCC and Cross-Correlation
Rahaman, M. E., Alam, S. M. S., Mondal, H. S., Muntaseer, A. S., Mandal, R., & Raihan, M. (2019) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt45670.2019.8944534 · Cited by 12
Huang, Y., Hou, H., Wang, Y., Zhang, Y., & Fan, M. (2020) · IEEE Access · doi:10.1109/access.2020.2974029 · Cited by 12
Li, S. C., Zhang, F., Yang, K., Liu, L., Liu, S., Hou, J., & Yi, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.10157 · Cited by 12
Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual LearningPreprint
Xu, X., Xie, H., Qin, S. J., Li, L., Tao, X., & Wang, F. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.01673 · Cited by 12
Wang, H., Cheng, M., Fu, Q., & Li, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095459 · Cited by 11
AudioVSR: Enhancing Video Speech Recognition with Audio Data
Yang, X., Cheng, X., Duan, J., Qiu, H., Hong, M., Fang, M., Ji, S., Zuo, J., Hong, Z., Zhang, Z., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.858 · Cited by 11
Enhancing Real-World Active Speaker Detection With Multi-Modal Extraction Pre-Training
Tao, R., Qian, X., Das, R. K., Gao, X., Wang, J., & Li, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521791 · Cited by 11
Tailored design of Audio–Visual Speech Recognition models using Branchformers
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101811 · Cited by 11
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
Rouditchenko, A., Thomas, S., Kuehne, H., Feris, R., & Glass, J. R. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3569210 · Cited by 11
A Comprehensive Survey of DeepFake Generation and Detection Techniques in Audio-Visual Media
Khan, I. N., Khan, K., & Ahmad, A. (2025) · ICCK Journal of Image Analysis and Processing · doi:10.62762/jiap.2025.431672 · Cited by 12
Hwang, J.-W., Park, J., Park, R., & Park, H. (2023) · Applied Acoustics · doi:10.1016/j.apacoust.2023.109478 · Cited by 11
Shashidhar, R., Shashank, M. P., & Sahana, B. (2023) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-023-08385-5 · Cited by 11
Laux, H., Mededovic, E., Hallawa, A., Märtin, L., Peine, A., & Schmeink, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448428 · Cited by 11
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation
Cheng, X., Huang, R., Li, L., Wang, Z., Jin, T., Yin, A., Chen, F., Duan, X., Huai, B., & Zhao, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.593 · Cited by 11
MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading
Zhang, W., Wang, J., Luo, Y., Yu, L., Yu, W., Zheng, H., & Shen, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i10.33104 · Cited by 11
Dynamic Facial Features in Positive-Emotional Speech for Identification of Depressive Tendencies
Liu, J., Huang, Y., Huang, X., Xia, X.-T., Niu, X.-X., Lin, L., & Chen, Y. (2020) · Smart innovation, systems and technologies · doi:10.1007/978-981-15-5852-8_12 · Cited by 11
Martín, A. G., Lara-Cabrera, R., Vargas, V. M., Gutiérrez, P. A., Hervás‐Martínez, C., & Camacho, D. (2020) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec48606.2020.9185914 · Cited by 11
A Closer Look at Audio-Visual Multi-Person Speech Recognition and Active Speaker Selection
Braga, O., & Siohan, O. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414160 · Cited by 11
An online intelligent electronic medical record system via speech recognition
Xia, X., Ma, Y., Luo, Y., & Lu, J. (2022) · International Journal of Distributed Sensor Networks · doi:10.1177/15501329221134479 · Cited by 11
MODEFORMER: Modality-Preserving Embedding For Audio-Video Synchronization Using Transformers
Gupta, A., Tripathi, R., & Jang, W. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097209 · Cited by 11
Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation
Li, L., Jin, T., Cheng, X., Wang, Y., Wang, L., Huang, R., & Zhao, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.699 · Cited by 11
Li, Z., Graave, T., Liu, J., Lohrenz, T., Kunzmann, S., & Fingscheidt, T. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389795 · Cited by 11
Semantically-Aware Attentive Neural Embeddings for Image-based Visual LocalizationPreprint
Seymour, Z., Sikka, K., Chiu, H.-P., Samarasekera, S., & Kumar, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03402 · Cited by 11
MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech RecognitionPreprint
Kim, S., Jang, K., Bae, S., Cho, S., & Yun, S.-Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.10447 · Cited by 11
Multimodal Learning Under Imperfect Data Conditions: A SurveyPreprint
Liaqat, M. I., Abbas, Q., Nawaz, S., Zaheer, Z., Moscati, M., Hou, Y., Khan, M. H., Khan, S. H., André, E., & Schedl, M. (2025) · doi:10.36227/techrxiv.176410566.65375877/v1 · Cited by 11
Audiovisual Speaker Separation with Full- and Sub-Band Modeling in the Time-Frequency Domain
Kalkhorani, V. A., Kumar, A., Tan, K., Xu, B., & Wang, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446297 · Cited by 10
Dong, P., Tian, S., Chen, S., Li, Y., Li, S., Zheng, M., & Yao, S. (2024) · Advanced Materials Technologies · doi:10.1002/admt.202400990 · Cited by 10
Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
Ma, Y., & Sun, X. N. (2025) · Applied Sciences · doi:10.3390/app15084142 · Cited by 10
UniSync: A Unified Framework for Audio-Visual Synchronization
Feng, T., Xie, Y., Guan, X., Song, J., Liu, Z., Ma, F., & Yu, F. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209535 · Cited by 10
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Cappellazzo, U., Kim, M., Ma, P., Chen, H., Liu, X., Petridis, S., & Pantić, M. (2025) · neural information processing systems · doi:10.52202/085713-1336 · Cited by 10
PAL: Prompting analytic learning with missing modality for multi-modal class-incremental learning
Yue, X., Chen, Y., Zhang, X., Gao, X., Feng, M., Lao, M., Zhuang, H., & Li, H. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113467 · Cited by 10
Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion
Lim, D., Kim, Y., Kim, D.-H., Yang, D.-H., & Chang, J.-H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434748 · Cited by 10
Whose emotion matters? Speaking activity localisation without prior knowledge
Carneiro, H., Weber, C., & Wermter, S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126271 · Cited by 10
PADVG: A Simple Baseline of Active Protection for Audio-Driven Video Generation
Liu, H., Liu, X., Tan, Z., Li, X., & Zhao, Y. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3638556 · Cited by 10
KMSAV: Korean multi‐speaker spontaneous audiovisual dataset
Park, K., Oh, C., & Dong, S. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0352 · Cited by 10
VoxMM: Rich Transcription of Conversations in the Wild
Kwak, D., Jung, J., Nam, K.-H., Jang, Y., Jung, J.-W., Watanabe, S., & Chung, J. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446300 · Cited by 10
Self-citationMultimodal Transformer Distillation for Audio-Visual Synchronization
Chen, X., Wu, H., Wang, C.-C., Lee, H.-Y., & Jang, J. R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446372 · Cited by 10
Text-Driven Talking Face Synthesis by Reprogramming Audio-Driven Models
Choi, J., Kim, M., Park, S. J., & Ro, Y. M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448506 · Cited by 10
Simulation-Driven Design of Smart Gloves for Gesture Recognition
Leite, C. F. S., Byvshev, P., Mauranen, H., & Xiao, Y. (2022) · Scientific Reports · doi:10.1038/s41598-024-65069-2 · Cited by 10
VividWav2Lip: High-Fidelity Facial Animation Generation Based on Speech-Driven Lip Synchronization
Liu, L., Wang, J., Chen, S., & Li, Z. (2024) · Electronics · doi:10.3390/electronics13183657 · Cited by 10
A Comprehensive Survey of Side-Channel Sound-Sensing Methods
Chen, Y., Yu, J., Kong, L., & Zhu, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3501334 · Cited by 10
Phonemes Convey Embodied Emotion
Yu, C. S. P., McBeath, M. K., & Glenberg, A. M. (2021) · Springer eBooks · doi:10.1007/978-3-030-78471-3_10 · Cited by 10
Improving Reverberant Speech Separation with Synthetic Room Impulse Responses
Aralikatti, R. C., Ratnarajah, A. J., Tang, Z., & Manocha, D. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688203 · Cited by 10
Spotting Visual Keywords from Temporal Sliding Windows
Yao, Y., Wang, T., Du, H., Zheng, L., & Gedeon, T. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3356101 · Cited by 10
End to End Lip Synchronization with a Temporal AutoEncoder
Shalev, Y., & Wolf, L. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093490 · Cited by 10
Efficient machine learning algorithm for electroencephalogram modeling in brain–computer interfaces
Yi, H. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04861-3 · Cited by 10
Audio Visual Speech Recognition using Feed Forward Neural Network Architecture
Shashidhar, R., Patilkulkarni, S., & Puneeth, S. B. (2020) · International Conference for Innovation in Technology (INOCON) · doi:10.1109/inocon50539.2020.9298429 · Cited by 10
Leveraging Visemes for Better Visual Speech Representation and Lip ReadingPreprint
Peymanfard, J., Saeedi, V., Mohammadi, M. R., Zeinali, H., & Mozayani, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10157 · Cited by 10
DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysisPreprint
Georgiou, E., Katsouros, V., Avrithis, Y., & Potamianos, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.11082 · Cited by 10
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.