Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 301–400 of 1,001
End-to-End Multi-Person Audio/Visual Automatic Speech Recognition
Braga, O., Makino, T., Siohan, O., & Liao, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053974 · Cited by 23
Indoor Scene Change Captioning Based on Multimodality Data
Qiu, Y., Satoh, Y., Suzuki, R., Iwata, K., & Kataoka, H. (2020) · Sensors · doi:10.3390/s20174761 · Cited by 23
Speaker-Independent Lipreading With Limited Data
Yang, C., Wang, S., Zhang, X., & Zhu, Y. H. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9190780 · Cited by 23
Visual Speech Enhancement Without A Real Visual Stream
Hegde, S. B., Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00197 · Cited by 23
MM-ALT: A Multimodal Automatic Lyric Transcription System
Gu, X., Ou, L., Ong, D., & Wang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548411 · Cited by 22
CATNet: Cross-modal fusion for audio–visual speech recognition
Wang, X., Mi, J., Li, B., Zhao, Y., & Meng, J. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.01.002 · Cited by 23
Chen, X., Wang, Y., Wu, X., Wang, D., Wu, Z., Liu, X., & Meng, H. M. L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445949 · Cited by 22
Mini-3DCvT: a lightweight lip-reading method based on 3D convolution visual transformer
Wang, H., Cui, B., Yuan, Q., Pu, G., Liu, X., & Zhu, J. (2024) · The Visual Computer · doi:10.1007/s00371-024-03515-y · Cited by 22
Synchronous Bidirectional Learning for Multilingual Lip Reading
Luo, M., Yang, S., Chen, X., Liu, Z., & Shan, S. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.172 · Cited by 22
Binaural audio generation via multi-task learning
Li, S., Liu, S., & Manocha, D. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480560 · Cited by 22
LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
Yemini, Y., Shamsian, A., Bracha, L., Gannot, S., & Fetaya, E. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2306.03258 · Cited by 22
Xu, X., Tu, W., & Yang, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102749 · Cited by 21
SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
Lin, Z., Zhao, Z., Li, H., Liu, J., Zhang, M., Zeng, X., & He, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475220 · Cited by 21
AVATAR: Unconstrained Audiovisual Speech Recognition
Gabeur, V., Seo, P. H., Nagrani, A., Sun, C., Alahari, K., & Schmid, C. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-776 · Cited by 21
Wu, X., Zhang, X., Feng, X., López, M. B., & Liu, L. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3220040 · Cited by 21
A Speech Recognition Method Based on Domain-Specific Datasets and Confidence Decision Networks
Dong, Z., Ding, Q., Zhai, W., & Zhou, M. (2023) · Sensors · doi:10.3390/s23136036 · Cited by 21
Towards Accurate Lip-to-Speech Synthesis in-the-Wild
Hegde, S. B., Mukhopadhyay, R., Jawahar, C. V., & Namboodiri, V. P. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611787 · Cited by 21
Jeevakumari, S. A. A., & Dey, K. (2024) · IEEE Access · doi:10.1109/access.2024.3436931 · Cited by 21
Audio-Driven Talking Face Generation with Stabilized Synchronization Loss
Yaman, D., Eyiokur, F. İ., Bärmann, L., Ekenel, H. K., & Waibel, A. H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72655-2_24 · Cited by 21
MAIN: Multimodal Attention-based Fusion Networks for Diagnosis Prediction
An, Y., Zhang, H., Sheng, Y., Wang, J., & Chen, X. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669634 · Cited by 21
End-to-End Lip-Reading Without Large-Scale Data
Fernandez-Lopez, A., & Sukno, F. M. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3182274 · Cited by 21
DARE: Deceiving Audio–Visual speech Recognition model
Mishra, S., Gupta, A. K., & Gupta, P. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107503 · Cited by 21
Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the WildPreprint
Wang, G., Zhang, P., Xie, L., Huang, W., & Zha, Y.-F. (2022) · arXiv.org · doi:10.48550/arxiv.2203.03984 · Cited by 18
MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face AnimationPreprint
Wu, H., Jia, J., Xing, J., Xu, H., Wang, X., & Wang, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09797 · Cited by 21
Visual Lip Reading Dataset in Turkish
Berkol, A., Sivri, T. T., Çolak, M., Akman, N. P., & Erdem, H. (2023) · Data · doi:10.3390/data8010015 · Cited by 20
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
Huang, R., Liu, H., Cheng, X., Ren, Y., Li, L., Ye, Z., He, J., Zhang, L., Liu, J., Yin, X., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.479 · Cited by 20
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition
Liu, L., Liu, L., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3363446 · Cited by 21
Multi-population evolutionary neural architecture search with stacked generalization
Song, C., Ma, Y., Yang, X., & Chen, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127664 · Cited by 21
STNet: Deep Audio–Visual Fusion Network for Robust Speaker Tracking
Li, Y., Liu, H., & Yang, B. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521737 · Cited by 20
CI-AVSR: A Cantonese Audio-Visual Speech Datasetfor In-car Command Recognition
Dai, W., Cahyawijaya, S., Yu, T., Barezi, E. J., Xu, P., Yiu, C. T. S., Frieske, R., Lovenia, H., Winata, G. I., Chen, Q., et al. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/23hae8o2v53q · Cited by 20
Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques
Seegehalli, P. J., & Niranjana Krupa, B. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103738 · Cited by 20
A multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset
Peymanfard, J., Heydarian, S., Lashini, A., Zeinali, H., Mohammadi, M. R., & Mozayani, N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121648 · Cited by 20
Scaling Multilingual Visual Speech Recognition
Prajwal, K. R., Hegde, S. B., & Zisserman, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890395 · Cited by 19
Self-citationUncovering the Visual Contribution in Audio-Visual Speech Recognition
Lin, Z., & Harte, N. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888423 · Cited by 19
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Cappellazzo, U., Kim, M., Petridis, S., Falavigna, D., & Brutti, A. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-111 · Cited by 19
Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild
Liang, C., Wang, Q., Chen, Y., & Tang, M. (2023) · Computer Animation and Virtual Worlds · doi:10.1002/cav.2226 · Cited by 19
MELDER: The Design and Evaluation of a Real-time Silent Speech Recognizer for Mobile Devices
Pandey, L., & Arif, A. S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642348 · Cited by 19
Liu, N., Wei, K., Yang, Y., Tao, J., Sun, X., Yao, F., Yu, H., Jin, L., Lv, Z., & Fan, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447778 · Cited by 19
Sheng, C., Pietikäinen, M., Tian, Q., & Liu, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475415 · Cited by 19
Versatile Backdoor Attack With Visible, Semantic, Sample-Specific and Compatible Triggers
Wang, R., Chen, H., Zhu, Z., Liu, L., & Wu, B. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3642161 · Cited by 19
Cross-Domain Deep Visual Feature Generation for Mandarin Audio–Visual Speech Recognition
Su, R., Liu, X., Wang, L., & Yang, J. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2950602 · Cited by 19
Zhou, Y., Zhang, Z., Wu, S., Jia, J., Jiang, Y., Sun, W., Liu, X., Min, X., & Zhai, G. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104321 · Cited by 18
Dementia Detection by Fusing Speech and Eye-Tracking Representation
Sheng, Z.-Y., Guo, Z., Li, X., Li, Y., & Ling, Z.-H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747054 · Cited by 18
Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs
Tao, R., Lee, K. A., Das, R. K., Hautamäki, V., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268568 · Cited by 18
CA-Wav2Lip: Coordinate Attention-based Speech To Lip Synthesis In The Wild
Wang, K.-C., Zhang, J., Huang, J., Li, Q. L., Sun, M.-T., Sakai, K., & Ku, W. (2023) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp58114.2023.00018 · Cited by 18
Li, T., Sun, Z., & Xiao, X. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3372469 · Cited by 18
Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey
Shahabaz, A., & Sarkar, S. (2024) · IEEE Access · doi:10.1109/access.2024.3391817 · Cited by 18
Speech Separation With Pretrained Frontend to Minimize Domain Mismatch
Wang, W., Pan, Z., Li, X., Wang, S., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3446242 · Cited by 18
PMMTalk$:$ Speech-Driven 3D Facial Animation From Complementary Pseudo Multi-Modal Features
Han, T., Gui, S., Huang, Y., Li, B., Liu, L., Zhou, B., Jiang, N., Lü, Q., Zhi, R., Liang, Y., et al. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521701 · Cited by 18
Using Deep Convolutional LSTM Networks for Learning Spatiotemporal Features
Courtney, L., & Sreenivas, R. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-41299-9_24 · Cited by 18
Ghorbani, S., Gaur, Y., Shi, Y., & Li, J. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383466 · Cited by 18
Fusing Information Streams in End-to-End Audio-Visual Speech Recognition
Yu, W., Zeiler, S., & Kolossa, D. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414553 · Cited by 18
Liu, Y., Wang, H., Song, K., Sun, M., Shao, Y., Xue, S., Li, L., Li, Y., Cai, H., Jiao, Y., et al. (2022) · Cancers · doi:10.3390/cancers14215181 · Cited by 18
CAD - Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00709 · Cited by 18
Visually Assisted Time-Domain Speech Enhancement
Ideli, E., Bruce, S., Bajić, I. V., & Vaughan, R. G. (2019) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip45357.2019.8969244 · Cited by 18
TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech SeparationPreprint
Xu, M., Li, K., Guo, C. H., & Hu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.01469 · Cited by 18
OMNI-AVSR: Towards Unified Multimodal Speech Recognition With Large Language Models
Cappellazzo, U., Liu, X., Ma, P., Petridis, S., & Pantić, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461431 · Cited by 17
Chen, X., Lin, W., Hua, J., Yao, L., Ding, Y., Li, B., Zeng, B., Shi, Y., Liu, Q., Zhang, Y., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37602-2_26 · Cited by 18
Egocentric Human Activities Recognition With Multimodal Interaction Sensing
Hao, Y., Kanezaki, A., Sato, I., Kawakami, R., & Shinoda, K. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3349191 · Cited by 17
Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models
Simic, C., & Bocklet, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448047 · Cited by 17
DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing
Sahipjohn, N., Gudmalwar, A. P., Shah, N. J., Wasnik, P., & Shah, R. R. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-1700 · Cited by 17
Speech Recognition Models are Strong Lip-readers
Prajwal, K. R., Afouras, T., & Zisserman, A. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2290 · Cited by 17
Self-citationSlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition
Wang, H., Kurita, S., Shimizu, S., & Kawahara, D. (2024) · Workshop on Advances in Language and Vision Research (ALVR) · doi:10.18653/v1/2024.alvr-1.11 · Cited by 17
Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising
Li, L., Cong, G., Qi, Y., Zha, Z.-J., Wu, Q., Sheng, Q. Z., Huang, Q., & Yang, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3597267 · Cited by 17
One-shot motion talking head generation with audio-driven model
Tang, P., Zhao, H., Meng, W., & Wang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129344 · Cited by 17
SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading
Wiriyathammabhum, P. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-63820-7_63 · Cited by 17
Audio–visual keyword transformer for unconstrained sentence‐level keyword spotting
Li, Y., Ren, J., Wang, Y., Wang, G., Li, X., & Liu, H. (2023) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12212 · Cited by 17
Rethinking the Visual Cues in Audio-Visual Speaker Extraction
Li, J., Ge, M., Pan, Z., Cao, R., Wang, L., Dang, J., & Zhang, S. S. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2545 · Cited by 17
Detection of Audio-Video Synchronization Errors Via Event Detection
Ebenezer, J. P., Wu, Y., Wei, H., Sethuraman, S., & Liu, Z. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414924 · Cited by 17
Contrastive Learning of Global-Local Video RepresentationsPreprint
Ma, S., Zeng, Z., McDuff, D., & Song, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05418 · Cited by 17
Cross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation
Yang, C.-C., Fan, W.-C., Yang, C.-F., & Wang, Y.-C. F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20210 · Cited by 16
“As for sentence-level speech recognition, previous works (Afouras et al. 2018; Chan et al. 2016; Garcia et al. 2019; Zhang, Cheng, and Wang 2019) aim at recognizing the character or word sequence in a full sentence by sequence-to-sequence learning.”
ResultImportance-Aware Information Bottleneck Learning Paradigm for Lip Reading
Sheng, C., Liu, L., Deng, W., Bai, L., Liu, Z., Lao, S., Kuang, G., & Pietikäinen, M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3210761 · Cited by 16
A robust adversarial attack against speech recognition with UAP
Qin, Z., Zhang, X., & Li, S. (2022) · High-Confidence Computing · doi:10.1016/j.hcc.2022.100098 · Cited by 16
Chandrabanshi, V., & Domnic, S. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03245-7 · Cited by 16
A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track
Alshahrani, M., & Maashi, M. (2024) · IEEE Access · doi:10.1109/access.2024.3404056 · Cited by 9
Audio-Visual Fusion With Temporal Convolutional Attention Network for Speech Separation
Liu, D., Zhang, T., Christensen, M. G., Yi, C., & An, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3463411 · Cited by 16
Kim, M., Yeo, J. H., Park, S. J., Rha, H., & Ro, Y. M. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681221 · Cited by 16
Che, N., Zhu, Y., Wang, H., Zeng, X., & Du, Q. (2024) · Applied Sciences · doi:10.3390/app15010199 · Cited by 16
Zhou, D., Zhang, Y., Wu, J., Zhang, X., Xie, L., & Yin, E. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3585165 · Cited by 16
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs
Cappellazzo, U., Kim, M., & Petridis, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434763 · Cited by 16
FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire
Liu, J., Ren, Y., Zhao, Z., Zhang, C., Huai, B., & Yuan, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413740 · Cited by 16
Poomhiran, L., Meesad, P., & Nuanmeesri, S. (2021) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.4102 · Cited by 16
Learning Contextually Fused Audio-Visual Representations For Audio-Visual Speech Recognition
Zhang, Z., Zhang, J., Zhang, J.-S., Wu, M., Fang, X., & Dai, L.-R. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897235 · Cited by 16
Relaxed Attention for Transformer Models
Lohrenz, T., Möller, B., Li, Z., & Fingscheidt, T. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191643 · Cited by 16
Audio-Visual Speech Recognition Using A Two-Step Feature Fusion Strategy
Liu, H., Xu, W., & Yang, B. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412454 · Cited by 16
Squeeze-and-Excitation network-Based Radar Object Detection With Weighted Location Fusion
Sun, P., Niu, X., Sun, P., & Xu, K. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463654 · Cited by 16
A Survey on Deep Multi-modal Learning for Body Language Recognition and GenerationPreprint
Liu, L., Gao, L., Lei, W., Ma, F., Lin, X., & Wang, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.08849 · Cited by 16
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation
Kim, S., Cho, S., Bae, S., Jang, K., & Yun, S.-Y. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2504.18539 · Cited by 16
An optimal 3D convolutional neural network based lipreading method
He, L., Ding, B., Wang, H., & Zhang, T. (2021) · IET Image Processing · doi:10.1049/ipr2.12337 · Cited by 15
Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition
Zuo, Y., Yao, H., Zhuang, L., & Xu, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387946 · Cited by 15
Raghuvanshi, D., Gao, X., Li, Z., Bansal, S., Coler, M., Kumar, N., & Nayak, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887864 · Cited by 15
Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language
Yeo, J. H., Kim, C. W., Kim, H., Rha, H., Han, S., Cheng, W.-H., & Ro, Y. M. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i9.33026 · Cited by 15
The gradient boosting method modification to forecast demand for individual types of goods
Leokhin, Y. L., Джалилевич, Ф. Т., & Zanegin, A. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-32-41 · Cited by 15
Deep Lip Reading-A Deep Learning Based Lip-Reading Software for the Hearing Impaired
Abrar, M. A., Islam, A. N. M. N., Hassan, M. M., Islam, M. T., Shahnaz, C., & Fattah, S. A. (2019) · IEEE Region Humanitarian Technology Conference · doi:10.1109/r10-htc47129.2019.9042439 · Cited by 15
Scale‐wise interaction fusion and knowledge distillation network for aerial scene recognition
Ning, H., Lei, T., An, M., Sun, H., Hu, Z., & Nandi, A. K. (2023) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12208 · Cited by 15
Research on Robust Audio-Visual Speech Recognition Algorithms
Yang, W., Li, P., Yang, W., Liu, Y., He, Y., Petrosian, O., & Davydenko, A. (2023) · Mathematics · doi:10.3390/math11071733 · Cited by 15
Robust Audio-Visual ASR with Unified Cross-Modal Attention
Li, J., Li, C., Wu, Y., & Qian, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096893 · Cited by 15
Axyonov, A., Ryumin, D., Ivanko, D., Kashevnik, A. M., & Karpov, A. A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448048 · Cited by 15
Audio-Visual Fusion And Conditioning With Neural Networks For Event Recognition
Brousmiche, M., Rouat, J., & Dupont, S. (2019) · Machine learning for signal processing ... · doi:10.1109/mlsp.2019.8918712 · Cited by 16
Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition
Yu, W., Zeiler, S., & Kolossa, D. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287841 · Cited by 15
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.