Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
Camgöz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01004 · Cited by 1,122
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
“ideos. Also, it should not be fine-tuned further on the generated frames like it is done in LipGAN. One such network that has been used to correct lip-sync errors for creating large lip-sync datasets [1, 3] is the SyncNet [9] model. We propose to adapt and train a modified version of SyncNet [9] for our task. 3.3.1 Ov…”
BackgroundMethodInfluentialHow, D. N. T., Hannan, M. A., Hossain Lipu, M. S., & Ker, P. J. (2019) · IEEE Access · doi:10.1109/access.2019.2942213 · Cited by 812
A General Survey on Attention Mechanisms in Deep Learning
Brauwers, G., & Frasincar, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3126456 · Cited by 695
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Shi, B., Hsu, W.-N., Lakhotia, K., & Mohamed, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.02184 · Cited by 110
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
Critical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Machine Learning Methods in Drug Discovery
Patel, L., Shukla, T., Huang, X., Ussery, D. W., & Wang, S. (2020) · Molecules · doi:10.3390/molecules25225277 · Cited by 460
Deep separable convolutional network for remaining useful life prediction of machinery
Wang, B., Lei, Y., Li, N., & Yan, T. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2019.106330 · Cited by 412
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 395
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation
Michelsanti, D., Tan, Z.-H., Zhang, S.-X., Xu, Y., Yu, M., Yu, D., & Jensen, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3066303 · Cited by 316
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475587 · Cited by 155
Lipreading Using Temporal Convolutional Networks
Martinez, B., Ma, P., Petridis, S., & Pantic, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053841 · Cited by 276
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., & Pantić, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096889 · Cited by 273
Survey on Deep Neural Networks in Speech and Vision Systems
Alam, M., Samad, M. D., Vidyaratne, L., Glandon, A., & Iftekharuddin, K. M. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.053 · Cited by 252
Towards Automatic Face-to-Face Translation
K R, P., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V., & Jawahar, C. V. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3351066 · Cited by 135
End-To-End Audio-Visual Speech Recognition with Conformers
Ma, P., Petridis, S., & Pantic, M. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414567 · Cited by 224
Qiao, M., Yan, S., Tang, X., & Xu, C. (2020) · IEEE Access · doi:10.1109/access.2020.2985617 · Cited by 219
Large-Scale Visual Speech Recognition
Shillingford, B., Assael, Y., Hoffman, M. W., Paine, T., Hughes, C., Prabhu, U., Liao, H., Sak, H., Rao, K., Bennett, L., et al. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1669 · Cited by 154
Self-citationSelf-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58523-5_13 · Cited by 217
Self-citationVideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild
Cheng, K., Cun, X., Zhang, Y., Xia, M., Yin, F., Zhu, M., Wang, X., Wang, J., & Wang, N. (2022) · SIGGRAPH Asia Conference Papers · doi:10.1145/3550469.3555399 · Cited by 217
Multi-channel Transformers for Multi-articulatory Sign Language Translation
Camgöz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66823-5_18 · Cited by 215
Deepfake Generation and Detection: A Benchmark and Survey
Pei, G., Zhang, J., Hu, M., Zhang, Z., Wang, C., Wu, Y., Zhai, G., Yang, J. J., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3801962 · Cited by 208
Data Augmentation for Deep Learning-Based Radio Modulation Classification
Huang, L., Pan, W., Zhang, Y., Qian, L., Gao, N., & Wu, Y. (2020) · IEEE Access · doi:10.1109/access.2019.2960775 · Cited by 202
You Said That?: Synthesising Talking Faces from Audio
Jamaludin, A., Chung, J. S., & Zisserman, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01150-y · Cited by 189
Self-citationSilent Speech Interfaces for Speech Restoration: A Review
González, J. A., Gomez-Alanis, A., Martín-Doñas, J. M., Pérez-Córdoba, J. L., & Gomez, A. M. (2020) · IEEE Access · doi:10.1109/access.2020.3026579 · Cited by 188
Robust Self-Supervised Audio-Visual Speech Recognition
Shi, B., Hsu, W.-N., & Mohamed, A. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-99 · Cited by 185
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Makino, T., Liao, H., Assael, Y., Shillingford, B., Garcia, B., Braga, O., & Siohan, O. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9004036 · Cited by 129
Missing data in multi-omics integration: Recent advances through artificial intelligence
Flores, J. E., Claborne, D. M., Weller, Z. D., Webb-Robertson, B.-J. M., Waters, K. M., & Bramer, L. M. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1098308 · Cited by 181
“For example, many of the deep-learning architectures mentioned in this review have similarities to ones used in analysis of digital media that try to combine multiple modalities to improve predictive performance of video content (Carreira and Zisserman, 2017; Afouras et al., 2019).”
Audio-Visual Speech Recognition with a Hybrid CTC/Attention Architecture
Petridis, S., Stafylakis, T., Ma, P., Tzimiropoulos, G., & Pantić, M. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639643 · Cited by 181
SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
Peng, Z., Hu, W., Shi, Y., Zhu, X., Zhang, X., Zhao, H., He, J., Liu, H., & Fan, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00070 · Cited by 179
BSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing Cues
Albanie, S., Varol, G., Momeni, L., Afouras, T., Chung, J. S., Fox, N., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_3 · Cited by 176
Self-citationDeep Audio-visual Learning: A Survey
Zhu, H., Luo, M.-D., Wang, R., Zheng, A.-H., & He, R. (2021) · International Journal of Automation and Computing · doi:10.1007/s11633-021-1293-0 · Cited by 174
Time Domain Audio Visual Speech Separation
Wu, J., Xu, Y., Zhang, S.-X., Chen, L., Yu, M., Xie, L., & Yu, D. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003983 · Cited by 171
A Multimodal Misinformation Detector for COVID-19 Short Videos on TikTok
Shang, L., Kou, Z., Zhang, Y., & Wang, D. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671928 · Cited by 171
ASR is All You Need: Cross-Modal Distillation for Lip Reading
Afouras, T., Chung, J. S., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054253 · Cited by 170
Self-citationVisual speech recognition for multiple languages in the wild
Ma, P., Petridis, S., & Pantic, M. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00550-z · Cited by 169
Yadav, S. P., Zaidi, S., Mishra, A., & Yadav, V. (2022) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09647-x · Cited by 169
DATA-GRU: Dual-Attention Time-Aware Gated Recurrent Unit for Irregular Multivariate Time Series
Tan, Q., Ye, M., Yang, B., Liu, S., Ma, A. J., Yip, T. C.-F., Wong, G. L.-H., & Yuen, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5440 · Cited by 121
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Gao, R., & Grauman, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01524 · Cited by 166
Sub-word Level Lip Reading With Visual Attention
Prajwal, K. R., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00510 · Cited by 164
Self-citationEmotional Speech-Driven Animation with Content-Emotion Disentanglement
Daněček, R., Chhatre, K., Tripathi, S., Wen, Y., Black, M. J., & Bolkart, T. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618183 · Cited by 160
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H. (2021) · arXiv · doi:10.48550/arxiv.2107.06592 · Cited by 156
Machine Learning Force Fields: Recent Advances and Remaining Challenges
Poltavsky, I., & Tkatchenko, A. (2021) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.1c01204 · Cited by 151
Towards Practical Lipreading with Distilled and Efficient Models
Ma, P., Martínez, B., Petridis, S., & Pantić, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415063 · Cited by 150
Multi-Modal Multi-Channel Target Speech Separation
Gu, R., Zhang, S.-X., Xu, Y., Chen, L., Zou, Y., & Yu, D. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2980956 · Cited by 146
Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition
Song, Q., Sun, B., & Li, S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3163771 · Cited by 145
Temporal Sentence Grounding in Videos: A Survey and Future Directions
Zhang, H., Sun, A., Jing, W., & Zhou, J. T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3258628 · Cited by 143
Shimada, K., Politis, A., Sudarsanam, P., Krause, D., Uchida, K., Adavanne, S., Hakala, A., Koyama, Y., Takahashi, N., Takahashi, S., et al. (2023) · neural information processing systems · doi:10.52202/075280-3189 · Cited by 143
Discriminative Multi-Modality Speech Recognition
Xu, B., Lu, C., Guo, Y., & Wang, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01444 · Cited by 137
EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis
Tan, S., Ji, B., Bi, M., & Pan, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72658-3_23 · Cited by 134
Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 128
Genetic programming for multiple-feature construction on high-dimensional classification
Tran, B., Xue, B., & Zhang, M. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.05.006 · Cited by 124
Joint Visual and Audio Learning for Video Highlight Detection
Badamdorj, T., Rochan, M., Wang, Y., & Cheng, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00802 · Cited by 123
My Lips Are Concealed: Audio-Visual Speech Enhancement Through Obstructions
Afouras, T., Chung, J. S., & Zisserman, A. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3114 · Cited by 122
Self-citationCochleaNet: A robust language-independent audio-visual model for real-time speech enhancement
Gogate, M., Dashtipour, K. K., Adeel, A., & Hussain, A. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.04.001 · Cited by 122
Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices
Ryumin, D., Ivanko, D., & Рюмина, Е. В. (2023) · Sensors · doi:10.3390/s23042284 · Cited by 119
Contrastive Positive Sample Propagation Along the Audio-Visual Event Line
Zhou, J., Guo, D., & Wang, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3223688 · Cited by 118
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116
“Dataset We train the proposed method in the train set (29 hours) of LRS2 and evaluate it with other competitive approaches in the test sets of two publicly available datasets: LRW [44], and LRS2 [45].”
MethodHearing Lips: Improving Lip Reading by Distilling Speech Recognizers
Zhao, Y., Xu, R., Wang, X., Hou, P., Tang, H., & Song, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6174 · Cited by 114
Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading
Zhang, X., Cheng, F., & Shilin, W. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00080 · Cited by 113
Stock Price Forecast Based on CNN-BiLSTM-ECA Model
Chen, Y., Fang, R., Liang, T., Sha, Z., Li, S., Yi, Y., Zhou, W., & Song, H. (2021) · Scientific Programming · doi:10.1155/2021/2446543 · Cited by 112
Audio- and Gaze-driven Facial Animation of Codec Avatars
Richard, A., Lea, C., Ma, S., Gall, J., la Torre, F. D., & Sheikh, Y. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00009 · Cited by 111
Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
Chen, Y., Xian, Y., Koepke, A. S., Shan, Y., & Akata, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00694 · Cited by 111
Stappen, L., Baird, A., Schumann, L., & Schuller, B. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3097002 · Cited by 91
Lip-reading with Densely Connected Temporal Convolutional Networks
Ma, P., Wang, Y., Shen, J., Petridis, S., & Pantić, M. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00290 · Cited by 109
Abdrakhmanova, M., Kuzdeuov, A., Jarju, S., Khassanov, Y., Lewis, M., & Varol, H. A. (2021) · Sensors · doi:10.3390/s21103465 · Cited by 107
Learning in Audio-visual Context: A Review, Analysis, and New PerspectivePreprint
Wei, Y., Hu, D., Tian, Y., & Li, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.09579 · Cited by 106
Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition
Zhang, Y., Yang, S., Xiao, J., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00134 · Cited by 105
Deep Learning-Based Automated Lip-Reading: A Survey
Fenghour, S., Chen, D., Guo, K., Li, B., & Xiao, P. (2021) · IEEE Access · doi:10.1109/access.2021.3107946 · Cited by 104
Audio-Visual Recognition of Overlapped Speech for the LRS2 Dataset
Yu, J., Zhang, S.-X., Wu, J., Ghorbani, S., Wu, B., Kang, S., Liu, S., Liu, X., Meng, H. M. L., & Yu, D. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054127 · Cited by 103
Identity-Preserving Talking Face Generation with Landmark and Appearance Priors
Zhong, W., Fang, C., Cai, Y., Wei, P., Zhao, G., Lin, L., & Li, G. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00938 · Cited by 102
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101
“Second, we use a generative model VQGAN [32], trained on LRS2 [3], for anomaly detection.”
MethodInfluentialDistinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
Kim, M., Yeo, J. H., & Ro, Y. M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.20003 · Cited by 101
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Ren, S., Du, Y., Lv, J., Han, G., & He, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01312 · Cited by 70
Recent Progress in the CUHK Dysarthric Speech Recognition System
Liu, S., Geng, M., Hu, S., Xie, X., Cui, M., Yu, J., Liu, X., & Meng, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3091805 · Cited by 96
Large Language Models are Strong Audio-Visual Speech Recognition Learners
Cappellazzo, U., Kim, M., Chen, H., Ma, P., Petridis, S., Falavigna, D., Brutti, A., & Pantić, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889251 · Cited by 96
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 95
Masked Contrastive Representation Learning for Reinforcement Learning
Zhu, J., Xia, Y., Wu, L., Deng, J., Zhou, W., Qin, T., Liu, T., & Li, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3176413 · Cited by 95
Hierarchical pattern recognition for tourism demand forecasting
Hu, M., Qiu, R. T. R., Wu, D. C., & Song, H. (2020) · Tourism Management · doi:10.1016/j.tourman.2020.104263 · Cited by 94
Hong, J. H., Kim, M., Choi, J., & Ro, Y. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01801 · Cited by 94
Audiovisual Generalised Zero-shot Learning with Cross-modal Attention and Language
Mercea, O.-B., Riesch, L., Koepke, A. S., & Akata, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01030 · Cited by 92
Audio-Visual Efficient Conformer for Robust Speech Recognition
Burchi, M., & Timofte, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00229 · Cited by 91
Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for LipreadingPreprint
Weng, X., & Kitani, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02540 · Cited by 91
Combining audio and visual speech recognition using LSTM and deep convolutional neural network
Shashidhar, R., Patilkulkarni, S., & Puneeth, S. B. (2022) · International Journal of Information Technology · doi:10.1007/s41870-022-00907-y · Cited by 90
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89
Anwar, M. A., Shi, B., Goswami, V., Hsu, W.-N., Pino, J., & Wang, C. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2279 · Cited by 89
Improving Multi-Modal Learning with Uni-Modal TeachersPreprint
Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11059 · Cited by 90
SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
Park, S. J., Kim, M., Hong, J., Choi, J., & Ro, Y. M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20102 · Cited by 88
Leo, M., Carcagnì, P., Mazzeo, P. L., Spagnolo, P., Cazzato, D., & Distante, C. (2020) · Information · doi:10.3390/info11030128 · Cited by 86
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference
Subedar, M., Krishnan, R., Meyer, P. L., Tickoo, O., & Huang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00640 · Cited by 85
A Survey of Multimodal Fake News Detection: A Cross-Modal Interaction Perspective
Li, X., Qiao, J., Yin, S., Wu, L., Gao, C., Wang, Z., & Li, X. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3543389 · Cited by 84
Muse: Multi-Modal Target Speaker Extraction with Visual Cues
Pan, Z., Tao, R., Xu, C., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414023 · Cited by 79
Jointly Learning Visual and Auditory Speech Representations from Raw DataPreprint
Haliassos, A., Ma, P., Mira, R., Petridis, S., & Pantić, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.06246 · Cited by 79
Serdyuk, D., Braga, O., & Siohan, O. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10920 · Cited by 73
Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation
Park, S. J., Kim, C., Rha, H., Kim, M., Hong, J. H., Yeo, J. H., & Ro, Y. M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.860 · Cited by 73
Multi-task Learning Approach for Automatic Modulation and Wireless Signal Classification
Jagannath, A., & Jagannath, J. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500447 · Cited by 72
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.