Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 101–200 of 1,001
Defending Multimodal Fusion Models against Single-Source Adversaries
Yang, K., Lin, W.-Y., Barman, M., Condessa, F., & Kolter, J. Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00335 · Cited by 71
LiRA: Learning Visual Speech Representations from Audio Through Self-Supervision
Ma, P., Mira, R., Petridis, S., Schuller, B. W., & Pantić, M. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1360 · Cited by 71
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition
Pan, X., Chen, P., Gong, Y., Zhou, H., Wang, X., & Lin, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.308 · Cited by 71
Application of convolutional neural networks in medical images: a bibliometric analysis
Jia, H., Zhang, J., Ma, K., Qiao, X., Ren, L., & Shi, X. (2024) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-23-1600 · Cited by 72
Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation
Lee, J., Chung, S.-W., Kim, S., Kang, H.-G., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00139 · Cited by 71
Selective Listening by Synchronizing Speech With Lips
Pan, Z., Tao, R., Xu, C., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153258 · Cited by 70
SepPrune: Structured Pruning for Efficient Deep Speech Separation
Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., Xu, H., Tian, Y., & Lu, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i38.40455 · Cited by 69
Deep transfer learning for conditional shift in regression
Liu, X., Li, Y., Meng, Q., & Chen, G. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107216 · Cited by 69
Zhu, Q., Zhou, L., Zhang, Z., Liu, S., Jiao, B., Zhang, J., Dai, L.-R., Jiang, D., Li, J., & Wei, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3275873 · Cited by 68
Yang, S., Wang, H., & Chen, B. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3329532 · Cited by 68
An efficient encoder-decoder architecture with top-down attention for speech separationPreprint
Li, K., Yang, R., & Hu, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15200 · Cited by 69
Fusion of tactile and visual information in deep learning models for object recognition
Babadian, R. P., Faez, K., Amiri, M., & Falotico, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.032 · Cited by 67
A Lip Reading Method Based on 3D Convolutional Vision Transformer
Wang, H., Pu, G., & Chen, T. (2022) · IEEE Access · doi:10.1109/access.2022.3193231 · Cited by 67
Siddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 67
Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization
Jiang, H., Murdock, C., & Ithapu, V. K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01029 · Cited by 66
Conformer is All You Need for Visual Speech Recognition
Chang, O., Liao, H., Serdyuk, D., Shah, A., & Siohan, O. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446532 · Cited by 65
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
BackgroundMethodInfluentialA Survey of Research on Lipreading Technology
Hao, M., Mamut, M., Yadikar, N., Aysa, A., & Ubul, K. (2020) · IEEE Access · doi:10.1109/access.2020.3036865 · Cited by 64
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN TrainingPreprint
Guan, L., Yin, W., Li, D., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04610 · Cited by 64
Audio-Visual Deep Neural Network for Robust Person Verification
Qian, Y., Chen, Z., & Wang, S. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3057230 · Cited by 63
“features to help speech recognition [14]–[16] or speech separation tasks [17]–[20].”
BackgroundMulti-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
Kim, M., Hong, J. H., Park, S. J., & Ro, Y. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00036 · Cited by 63
Sleep staging by bidirectional long short-term memory convolution neural network
Chen, X., He, J. Q., Wu, X., Yan, W., & Wei, W. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.03.019 · Cited by 62
Chen, H., Zhou, H., Du, J., Lee, C., Chen, J., Watanabe, S., Siniscalchi, S. M., Scharenborg, O., Liu, D.-Y., Yin, B., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746683 · Cited by 61
Jagannath, A., & Jagannath, J. (2022) · Physical Communication · doi:10.1016/j.phycom.2022.101793 · Cited by 61
Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
Luo, M., Yang, S., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00010 · Cited by 61
A Deep Fourier Residual method for solving PDEs using Neural Networks
Taylor, J. M., Pardo, D., & Muga, I. (2022) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2022.115850 · Cited by 60
Imaginary Voice: Face-Styled Diffusion Model for Text-to-Speech
Lee, J.-Y., Chung, J. S., & Chung, S.-W. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094745 · Cited by 60
Self-citationSeeing wake words: Audio-visual Keyword Spotting
Momeni, L., Afouras, T., Stafylakis, T., Albanie, S., & Zisserman, A. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.10 · Cited by 59
Self-citationLian, J., Baevski, A., Hsu, W.-N., & Auli, M. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389642 · Cited by 58
Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning
Li, W., Wang, P., Xiong, R., & Fan, X. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3430080 · Cited by 58
Lipreading with DenseNet and resBi-LSTM
Chen, X., Du, J., & Zhang, H. (2020) · Signal Image and Video Processing · doi:10.1007/s11760-019-01630-1 · Cited by 57
Multimodal Attention-Mechanism For Temporal Emotion Recognition
Ghaleb, E., Niehues, J., & Asteriadis, S. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191019 · Cited by 57
Neural Dubber: Dubbing for Videos According to ScriptsPreprint
Hu, C., Tian, Q., Li, T., Wang, Y.-P., Wang, Y., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08243 · Cited by 57
On quadrature rules for solving Partial Differential Equations using Neural Networks
Rivera, J. A., Taylor, J. M., Omella, Á. J., & Pardo, D. (2022) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2022.114710 · Cited by 56
Wang, J., Zhao, K., Zhang, S., Zhang, Y., Shen, Y., Zhao, D., & Zhou, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01330 · Cited by 56
M^4I: Multi-Modal Models Membership Inference
Hu, P., Wang, Z., Sun, R., Wang, H., & Xue, M. (2022) · neural information processing systems · doi:10.52202/068431-0136 · Cited by 57
Wu, L., Lin, X., Chen, Z., Lin, P., & Cheng, S. (2021) · Structural Control and Health Monitoring · doi:10.1002/stc.2766 · Cited by 56
Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)Preprint
Chung, J. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10555 · Cited by 56
Self-citationDeep Person Generation: A Survey from the Perspective of Face, Pose, and Cloth Synthesis
Sha, T., Zhang, W., Shen, T., Li, Z., & Mei, T. (2023) · ACM Computing Surveys · doi:10.1145/3575656 · Cited by 55
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Semantic and Relation Modulation for Audio-Visual Event Localization
Wang, H., Zha, Z.-J., Li, L., Chen, X., & Luo, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3226328 · Cited by 54
An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
Li, K., Xie, F., Chen, H., Yuan, K., & Hu, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3384034 · Cited by 54
Lip to Speech Synthesis with Visual Context Attentional GANPreprint
Kim, M., Hong, J. H., & Ro, Y. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.01726 · Cited by 54
Chen, C., Li, R., Hu, Y., Siniscalchi, S. M., Chen, P.-Y., Chng, E., & Yang, C.-H. H. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2402.05457 · Cited by 54
Backdooring Multimodal Learning
Han, X., Wu, Y., Zhang, Q., Zhou, Y. H., Xu, Y., Qiu, H., Xu, G., & Zhang, T. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00031 · Cited by 53
Multimodal and Multiresolution Speech Recognition with Transformers
Paraskevopoulos, G., Parthasarathy, S., Khare, A., & Sundaram, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.216 · Cited by 53
Look&listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement
Xiong, J., Yu, Z., Zhang, P., Xie, L., Huang, W., & Zha, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3199109 · Cited by 52
A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition
Ivanko, D., Ryumin, D., & Karpov, A. A. (2023) · Mathematics · doi:10.3390/math11122665 · Cited by 52
SynthVSR: Scaling Up Visual Speech RecognitionWith Synthetic Supervision
Liu, X., Lakomkin, E., Vougioukas, K., Ma, P., Chen, H., Xie, R., Doulaty, M., Moritz, N., Kolář, J., Petridis, S., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01803 · Cited by 52
Tal: A Synchronised Multi-Speaker Corpus of Ultrasound Tongue Imaging, Audio, and Lip Videos
Ribeiro, M. S., Sanger, J., Zhang, J.-X., Eshky, A., Wrench, A., Richmond, K., & Renals, S. J. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383619 · Cited by 52
Listening to Sounds of Silence for Speech DenoisingPreprint
Xu, R., Wu, R., Ishiwaka, Y., Vondrick, C., & Zheng, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12013 · Cited by 52
Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy
Li, D., Gao, Y., Zhu, C., Wang, Q., & Wang, R. (2023) · Sensors · doi:10.3390/s23042053 · Cited by 51
Artificial intelligence: A key fulcrum for addressing complex environmental health issues
Huang, L., Duan, Q., Liu, Y., Wu, Y., Li, Z., Guo, Z., Liu, M., Lu, X., Wang, P., Liu, F., et al. (2025) · Environment International · doi:10.1016/j.envint.2025.109389 · Cited by 51
Audio-Visual Synchronisation in the wild
Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., & Zisserman, A. (2021) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2112.04432 · Cited by 50
Self-citationTVLT: Textless Vision-Language Transformer
Tang, Z., Cho, J., Nie, Y., & Bansal, M. (2022) · neural information processing systems · doi:10.52202/068431-0699 · Cited by 50
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
Qu, L., Weber, C., & Wermter, S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3191677 · Cited by 49
Kim, M., Yeo, J. H., Choi, J., & Ro, Y. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01409 · Cited by 49
Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading
Sheng, C., Zhu, X., Xu, H., Pietikäinen, M., & Liu, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3102433 · Cited by 49
Ryumin, D., Axyonov, A., Рюмина, Е. В., Ivanko, D., Kashevnik, A. M., & Karpov, A. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124159 · Cited by 48
Jagannath, J., Jagannath, A., Furman, S., & Gwin, T. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-77939-9_2 · Cited by 48
LipFormer: Learning to Lipread Unseen Speakers Based on Visual-Landmark Transformers
Xue, F., Yu, L., Liu, D., Xie, Y., Wu, L. Y., & Hong, R. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3282224 · Cited by 47
CroMM-VSR: Cross-Modal Memory Augmented Visual Speech Recognition
Kim, M., Hong, J. H., Park, S. J., & Ro, Y. M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3115626 · Cited by 47
Towards Realistic Visual Dubbing with Heterogeneous Sources
Xie, T., Liao, L., Bi, C., Tang, B., Yin, X., Yang, J., Wang, M., Yao, J., Zhang, Y., & Ma, Z. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475318 · Cited by 47
Hong, J. H., Kim, M., Yoo, D., & Ro, Y. M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11311 · Cited by 47
Automatic modulation recognition using CNN deep learning models
Mohsen, S., Ali, A. M., & Emam, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15814-y · Cited by 46
In‐Sensor Touch Analysis for Intent Recognition
Xu, Y., Yu, S., Liu, L., Lin, W., Cao, Z., Hu, Y., Duan, J., Huang, Z., Wei, C., Guo, Z., et al. (2024) · Advanced Functional Materials · doi:10.1002/adfm.202411331 · Cited by 46
Jointly Learning of Visual and Auditory: A New Approach for RS Image and Audio Cross-Modal Retrieval
Mao, G., Zhou, C., & Liu, J. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2949220 · Cited by 46
Alayba, A. M., & Palade, V. (2021) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.12.004 · Cited by 46
CN-CVS: A Mandarin Audio-Visual Dataset for Large Vocabulary Continuous Visual to Speech Synthesis
Chen, C., Wang, D., & Zheng, T. F. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095796 · Cited by 45
Hyperbolic Audio-visual Zero-shot Learning
Hong, J., Hayder, Z., Han, J., Fang, P., Harandi, M. T., & Petersson, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00724 · Cited by 45
Audio-Visual Speech Recognition is Worth $32\times 32\times 8$ Voxels
Serdyuk, D., Braga, O., & Siohan, O. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688191 · Cited by 45
Intelligible Lip-to-Speech Synthesis with Speech Units
Choi, J., Kim, M., & Ro, Y. M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-194 · Cited by 44
SlideSpeech: A Large Scale Slide-Enriched Audio-Visual Corpus
Wang, H., Yu, F., Shi, X., Wang, Y., Zhang, S. S., & Li, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448079 · Cited by 43
Exposing Lip-syncing Deepfakes from Mouth Inconsistencies
Datta, S. K., Jia, S., & Lyu, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687902 · Cited by 45
A Decade of Video Analytics at Edge: Training, Deployment, Orchestration, and Platforms
Bai, T., Zhao, H., Huang, L., Wang, Z., Kim, D. I., & Nallanathan, A. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3563377 · Cited by 43
Automatic Classification of Text Complexity
Santucci, V., Santarelli, F., Forti, L., & Spina, S. (2020) · Applied Sciences · doi:10.3390/app10207285 · Cited by 43
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
Patrick, M., Huang, P.-Y., Misra, I., Metze, F., Vedaldi, A., Asano, Y. M., & Henriques, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01039 · Cited by 43
Lip-to-Speech Synthesis in the Wild with Multi-Task Learning
Kim, M., Hong, J. H., & Ro, Y. M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095582 · Cited by 43
Modality Dropout for Improved Performance-driven Talking Faces
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418840 · Cited by 42
Dai, Y., Chen, H., Du, J., Wang, R., Chen, S., Wang, H., & Lee, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02591 · Cited by 42
SoundLip: Enabling Word and Sentence-level Lip Interaction for Smart Devices
Zhang, Q., Wang, D., Zhao, R., & Yu, Y. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3448087 · Cited by 42
Prompt Tuning of Deep Neural Networks for Speaker-Adaptive Visual Speech Recognition
Kim, M., Kim, H.-I., & Ro, Y. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3484658 · Cited by 41
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens
Yeo, J. H., Rha, H., Park, S. J., & Ro, Y. M. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1065 · Cited by 41
Improving Multimodal Learning via Imbalanced Learning
Wei, S., Luo, C., & Luo, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00217 · Cited by 41
How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2980436 · Cited by 41
Speech Emotion Recognition Based on Robust Discriminative Sparse Regression
Song, P., Zheng, W., Yu, Y., & Ou, S. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.2990928 · Cited by 41
Every node counts: Self-ensembling graph convolutional networks for semi-supervised learning
Luo, Y., Ji, R., Guan, T., Yu, J., Liu, P., & Yang, Y. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107451 · Cited by 41
Towards Pose-Invariant Lip-Reading
Cheng, S., Ma, P., Tzimiropoulos, G., Petridis, S., Bulat, A., Shen, J., & Pantić, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054384 · Cited by 41
mSilent: Towards General Corpus Silent Speech Recognition Using COTS mmWave Radar
Zeng, S., Wan, H., Shi, S., & Wang, W. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3580838 · Cited by 40
EMMN: Emotional Motion Memory Network for Audio-driven Emotional Talking Face Generation
Tan, S., Ji, B., & Pan, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02024 · Cited by 39
“However, the Wav2Lip model is trained on LRS2 [2] and there exists an inevitable gap be-tween LRS2 and MEAD, resulting in slightly poor performance in MEAD.”
MethodLiu, M., Wang, J., Qian, X., & Li, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3326694 · Cited by 39
Jiang, S., & Chen, Z. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21625 · Cited by 39
Robust Noisy Correspondence Learning with Equivariant Similarity Consistency
Yang, Y., Wang, L., Yang, E., & Deng, C. D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01676 · Cited by 40
Cross-Modal Knowledge Distillation With Multi-Stage Adaptive Feature Fusion for Speech Separation
Fan, C., Xiang, W., Tao, J., Yi, J., & Lv, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533359 · Cited by 39
Machine Learning Techniques used for the Histopathological Image Analysis of Oral Cancer-A Review
Panigrahi, S., & Swarnkar, T. (2020) · The Open Bioinformatics Journal · doi:10.2174/1875036202013010106 · Cited by 39
HLR-Net: A Hybrid Lip-Reading Model Based on Deep Convolutional Neural Networks
Sarhan, A. M., Elshennawy, N. M., & Ibrahim, D. M. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2021.016509 · Cited by 39
VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices
Kadandale, V. S., Montesinos, J. F., & Haro, G. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10861 · Cited by 39
Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition
Hu, Y., Li, R., Chen, C., Zou, H., Zhu, Q., & Chng, E. S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/564 · Cited by 39
The Right to Talk: An Audio-Visual Transformer Approach
Truong, T.-D., Duong, C. N., De Vu, T., Pham, H. A., Raj, B., Le, N., & Luu, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00114 · Cited by 38
“To study the effectiveness of our proposed cross-domain alignment method, we employ an ablation study with audio separation task on LRS2 using two configurations: without and with Lalign.”
MethodInfluentialUnified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond
Li, J., Li, C., Wu, Y., & Qian, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3375641 · Cited by 38
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.