Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 801–900 of 1,001
Selective Hearing through Lip-readingPreprint
Pan, Z., Tao, R., Xu, C., & Li, H. (2021) · arXiv (Cornell University) · Cited by 2
Deep Cross-Modal Alignment in Audio-Visual Speech Recognition
Sterpu, G. (2021) · Trinity's Access to Research Output (TARA) (Trinity College Dublin) · Cited by 2
Improving Reverberant Speech Separation with Multi-stage Training and Curriculum LearningPreprint
Aralikatti, R. C., Ratnarajah, A. J., Tang, Z., & Manocha, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09177 · Cited by 2
Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip ReadingPreprint
Elashmawy, S., Ramsis, M., Eraqi, H. M., Eldeshnawy, F., Mabrouk, H., Abugabal, O., & Sakr, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.03543 · Cited by 2
Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car CommandsPreprint
Dai, W., Cahyawijaya, S., Yu, T., Barezi, E. J., & Fung, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.02663 · Cited by 2
Mabrouk, H., Abugabal, O., Sakr, N., & Eraqi, H. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.05692 · Cited by 2
Automated Word-level Lip Reading using Convolutional Neural Networks with New Turkish DatasetPreprint
Akman, N. P., Berkol, A., & Erdem, H. (2023) · Preprints.org · doi:10.20944/preprints202304.0645.v1 · Cited by 2
Yeo, J. H., Rha, H., Park, S., Won, J., & Ro, Y. M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.20476 · Cited by 2
CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative SynchronizationPreprint
Bai, D., Ma, Z., Wei, X., & Bo, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.03186 · Cited by 2
DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And IntelligibilityPreprint
Liu, Y., Fang, Y., & Lin, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05223 · Cited by 2
Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity PreservationPreprint
Yaman, D., Eyiokur, F. İ., Bärmann, L., Ekenel, H. K., & Waibel, A. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.20953 · Cited by 2
Multi-Modal Learning for Video Understanding
Gabeur, V. (2022) · theses.fr (ABES) · doi:10.70675/2b02ff4fzec66z4610z8f97za1abfa8705ff · Cited by 2
SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference LearningPreprint
Cheng, X., Wang, X., Ba, Y., Wang, Y., Guan, K., Wang, Y., Li, W., & Song, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.12179 · Cited by 2
Su, F., Li, C., Li, M., & Liu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.05763 · Cited by 2
SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex ScenariosPreprint
Jiang, Z., Chen, Y., Pan, Z., Qian, X., Xing, B., Tsang, I. W.-H., Yin, X.-C., & Li, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.02343 · Cited by 2
Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic ScenesPreprint
Sang, W., Li, K., Li, Y., Huang, J., & Hu, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.32749 · Cited by 2
Anshul, A., Gopal, S., Rajan, D., & Chng, E. S. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01283 · Cited by 1
[Research progress on electronic health records multimodal data fusion based on deep learning].
(2024) · PubMed Central · doi:10.7507/1001-5515.202310011 · Cited by 1
Patil, S. A., Dhoble, V., Gawade, S., Jagdale, P., & Jinde, R. (2022) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-2874 · Cited by 1
Audio-to-Facial Landmarks Generator for Talking Face Video Synthesis
Jeong, D., Lee, I., & Paik, J. (2023) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic57457.2023.10049847 · Cited by 1
Cycle-Consistent Generative Adversarial Network Architectures for Audio Visual Speech Recognition
He, Y., Seng, K. P., Ang, L.-M., & Zhao, X. (2023) · IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC) · doi:10.1109/icspcc59353.2023.10400358 · Cited by 1
Relational-branchformer: Novel framework for audio-visual speech recognition
Xiao, Y., Liu, X., Zhu, A., & Huang, J. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105182 · Cited by 1
Plug-and-play learnable EMD module for time-domain speech separation
Yang, X., Ke, W., & Wei, J. (2024) · Applied Acoustics · doi:10.1016/j.apacoust.2024.110210 · Cited by 1
Hide and Recognize Your Privacy Image
Zhu, Z., Zhou, H., Hu, H., Jiang, Q., Qian, Z., & Zhang, X. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3456103 · Cited by 1
Adversarial Purification by Consistency-Aware Latent Space Optimization on Data Manifolds
Zhang, S., Yang, J., Luo, H., Chen, J., Wang, K., Liu, F., Han, B., & Tan, M. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3720823 · Cited by 1
Lip-Reading Classification of Turkish Digits Using Ensemble Learning Architecture Based on 3DCNN
Erbey, A., & Barışçı, N. (2025) · Applied Sciences · doi:10.3390/app15020563 · Cited by 1
Summary of Low-Resource Dysarthria Wake-Up Word Spotting Challenge
Gao, M., Chen, H., Du, J., Xu, X., Guo, H., Bu, H., Li, M., & Lee, C. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832239 · Cited by 1
Visually-guided Order-fixed Speech Separation Algorithm
Shi, D., Deng, Y., & Wei, Y. (2024) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc61627.2024.10837259 · Cited by 1
Li, Y., Zhou, X., He, Z., Yu, W., & Luo, Y. H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888235 · Cited by 1
Expressive talking face generation via audio visual control
Li, P., Zhao, H., Lin, M., Liu, Q., Tang, P., & Zhou, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01759-9 · Cited by 1
Talking Face Generation With Lip and Identity Priors
Wu, J., Li, F. W. B., Tam, G. K. L., Yang, B., Nan, F., & Pan, J. (2025) · Computer Animation and Virtual Worlds · doi:10.1002/cav.70026 · Cited by 1
Xiao, Z., Liu, X., Guo, J., He, J., Hong, R., & Wang, M. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3742790 · Cited by 1
A Planer Moving Microphone Array for Sound Source Localization
Wang, C., Chu, H. K., & Choy, Y. S. (2025) · Applied Sciences · doi:10.3390/app15126777 · Cited by 1
Hardware Friendly Transformer Optimization with Dynamic Attention Matrix Fusion
Yang, Q., Wang, X., Zhou, Y., Li, Q., & Qiao, S. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11044099 · Cited by 1
Negative Adjustment for Contrastive Learning in Audio-Visual Generalized Zero-Shot Learning
Wen, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6951-6_1 · Cited by 1
Lip Enhancement and Multi-View Simulation for Robust Visual Speech Recognition in MAVSR 2025
Su, F., Li, C., & Liu, J. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099112 · Cited by 1
Evaluating Lip-Reading Approaches: Comparative Study of Recent Techniques
Gamal, H., Elsayad, D., Amin, S. E.-S., & Ebied, H. M. (2025) · International Conference on Machine Intelligence and Smart Innovation (ICMISI) · doi:10.1109/icmisi65108.2025.11115312 · Cited by 1
Chen, Z., Xu, C., Ahmad, H., Okadome, Y., Ishiguro, H., & Nakamura, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3602885 · Cited by 1
Geldhauser, C., Liljegren, J., & Nordqvist, P. (2025) · Electronics · doi:10.3390/electronics14173487 · Cited by 1
MC-Mamba: Cross-modal target speaker extraction model based on multiple consistency
Lv, K., Deng, Y., & Wei, Y. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103304 · Cited by 1
Rajan, R., Thakshinraj, S., Raghav, J. K., Roosevelt, S. J., Pappa, D. C., Rajest, S. S., Singh, B., & Singh, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1987-2.ch007 · Cited by 1
Liu, K., Wei, J., He, S., Ma, Z., Zhang, C., Xie, N., & Yang, Y. (2025) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2025/176 · Cited by 1
Research of load balancing in three-channel cluster systems with adaptive dispatching
Goncharenko, V. A., Хомоненко, А. Д., & Khalil, M. M. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-19-31 · Cited by 1
Laksana, I. K. P. B., Indrawan, J. O., Lucky, R. V., & Lucky, H. (2025) · doi:10.1109/icoict66265.2025.11192894 · Cited by 1
Vision-Integrated High-Quality Neural Speech Coding
Guo, Y., Ai, Y., Zheng, R.-C., Du, H.-P., Jiang, X.-H., & Ling, Z.-H. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1409 · Cited by 1
MISP-QEKS: A Large-Scale Dataset with Multimodal Cues for Query-by-Example Keyword Spotting
Xiong, S., Chen, H., Cheng, S., Shen, K., Zhou, H., Wan, G., Zhang, C.-Y., Li, K., Du, J., & Dai, L.-R. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758268 · Cited by 1
Design and implementation of metaheuristic-driven BiLSTM hybrid model for silent communication
Sonawane, S., & Malathi, P. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107914 · Cited by 1
Multi-modal Speech Recognition: Integrating Audio and Visual Information with Deep Neural Network
Pathak, N., Shrestha, G. P., Vijh, S., & Kumar, S. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-2577-5_34 · Cited by 1
Su, F., Li, C., & Liu, J. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209403 · Cited by 1
Domain adaptative keyword spotting with multimodal enhancement
Chen, L., & Wang, H. (2025) · Multimedia Systems · doi:10.1007/s00530-025-02091-y · Cited by 1
VAPO: End-to-end Slide-Enhanced Speech Recognition with Omni-modal Large Language Models
Hu, R., Qiu, D., Wang, Y., Liu, S., & Sang, J. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.425 · Cited by 1
Learning Speaker-Invariant Visual Features for Lipreading
Li, Y., Xue, F., Li, S., Zhang, J., Guo, D., Liu, Z., & Hong, R. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3821639 · Cited by 1
Cross-Modal Meta-Generalization for Silent Speech Recognition via Ultrasound Tongue Images
Guo, M., Fang, Q., Zhang, H., Lu, W., & Wei, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3646808 · Cited by 1
Prem, N., & Lijiya, A. (2025) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct65861.2025.11306806 · Cited by 1
Event-based Lip Reading with Triplane Fusion Network
Ju, H., Zheng, Z., Xu, Z., Chen, W., Wang, L., Wang, D., Lu, H., & Jia, X. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3787856 · Cited by 1
Su, F., Li, C., Liu, J., Ju, W., Suo, H., & (91180), M. L. (2026) · Interspeech (USB) · doi:10.21437/interspeech.2026-1277 · Cited by 1
Divya, & Suresha, D. (2026) · International Journal of Image Graphics and Signal Processing · doi:10.5815/ijigsp.2026.02.06 · Cited by 1
Fu, D., Cheng, X., Chen, J., Jin, T., & Zhang, Z. M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3682022 · Cited by 1
Charuau, D., & Harte, N. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.101990 · Cited by 1
Cross-Modal Bottleneck Fusion for Noise Robust Audio-Visual Speech Recognition
Ok, S., Choi, M. J., Kim, E., Han, S., & Lee, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462159 · Cited by 1
Wu, L., Zhang, X., Yuan, H., Zhang, Y., Zheng, C., Xie, L., Liu, T., & Yin, E. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460922 · Cited by 1
OCR-Enhanced Multimodal ASR Can Read While Listening
Chen, J., Tang, C., Li, Y., Sun, G., & Zhang, C. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463873 · Cited by 1
Science Tokyo CHiME-9 MCoRec System Description
Hartanto, R., Nitsu, D., Dinh, N. M. P., & Shinoda, K. (2026) · doi:10.21437/chime.2026-5 · Cited by 1
The USTC-NERCSLIP Systems for the CHiME-9 MCoRec Challenge
Jiang, Y., Wang, R., Zhang, J., Du, J., Han, Y., Quan, Z., Chen, H., Yang, Y., Zheng, K., Chen, Z., et al. (2026) · doi:10.21437/chime.2026-1 · Cited by 1
The SUSTech AILab System Description for CHiME-9 MCoRec Challenge
Ling, T., Shen, P., & Wang, Z.-Q. (2026) · doi:10.21437/chime.2026-3 · Cited by 1
A Review of Lip-Reading: From Datasets and Architectures to Cross-Linguistic Performance Gaps
Hu, Q., Kipli, K., Zulcaffle, T. M. A., Liu, Y., & Cao, J. (2026) · Advances in Technology Innovation · doi:10.46604/aiti.2026.16377 · Cited by 1
Wang, L., Xing, J., Hu, T., Jiang, J., Zhao, J., & Ma, H. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1381 · Cited by 1
VisG AV-HuBERT: Viseme-Guided AV-HuBERT
Papadopoulos, A., Jain, R., & Harte, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31335-5_45 · Cited by 1
CNN-Based Spatiotemporal Feature Extraction for Video Processing: A Systematic Review
Jimenez, A. E. L., Jiménez‐Hernández, H., Herrera-Navarro, A. M., Cantón-Enriquez, D., Hernández-Alvarado, R., Pérez-Ramos, J.-L., Hernández, A. G. M., & Ávila, J. C. M. (2026) · Electronics · doi:10.3390/electronics15163736 · Cited by 1
TGTSE: Token-Guided Target Speaker Extraction with Visual Cue
Ling, T., He, S., & Wang, Z.-Q. (2026) · Interspeech (USB) · doi:10.21437/interspeech.2026-1710 · Cited by 1
Deep Audio-visual System for Closed-set Word-level Speech Recognition
Yuan, Y., Tang, W., Fan, M., Cao, Y., Zhang, P., & Xie, L. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3356102 · Cited by 1
One Perceptron to Rule Them All: Language, Vision, Audio and Speech
Giró-i-Nieto, X. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3390740 · Cited by 1
Analysis of Audio Visual Feature Extraction Techniques for AVSR System
Kumar, L. A., Karthika Renuka, D., & Shunmuga Priya, M. C. (2021) · International Conference on Combinatorial and Optimization (ICCAP) · doi:10.4108/eai.7-12-2021.2314528 · Cited by 1
An Interference-Resistant and Low-Consumption Lip Recognition Method
Jia, J., Wang, Z., Xu, L., Dai, J., Gu, M., & Huang, J. (2022) · Electronics · doi:10.3390/electronics11193066 · Cited by 1
Emotional Talking Faces: Making Videos More Expressive and Realistic
Goyal, S., Uppal, S., Bhagat, S., Goel, D., Mali, S., Yu, Y., Yin, Y., & Shah, R. R. (2022) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3551626.3564976 · Cited by 1
A Comprehensive Review on Machine Learning and Deep Learning Methods in Drug Discovery
Khaire, S., & Bhaladhare, P. (2022) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v10i10.5728 · Cited by 1
Development of Small Vocabulary Continuous Speech-to-Text System for Kannada Language/Dialects
Thimmaraja Yadava, G., Nagaraja, B. G., Kumaran, S. Y., Ramachandra, A. C., & Kumar, N. M. A. (2022) · Indian Journal of Science and Technology · doi:10.17485/ijst/v15i45.1884 · Cited by 1
Gabor-based Audiovisual Fusion for Mandarin Chinese Speech Recognition
Xu, Y., Wang, H., Dong, Z., Li, Y., & Abel, A. K. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909634 · Cited by 1
Voice Keyword Spotting on Edge Devices
Saifullah, K., Quaiser, R. M., & Akhtar, N. (2022) · International Conference on Multimedia, Signal Processing and Communication Technologies (IMPACT) · doi:10.1109/impact55510.2022.10029228 · Cited by 1
Literature Review of Audio-Driven 2D Avatar Video Generation Algorithms
Li, Y., Zhang, H., Cao, S., Jiang, D., Wang, M., & Wang, W. (2023) · Lecture notes in operations research · doi:10.1007/978-981-99-3618-2_9 · Cited by 1
Lip Reading and Reconstruction using ML
Kuriakose, L. K., Joseph, M. R., Namita, R., Nabi, S., & Lone, T. A. (2023) · International Journal on Emerging Research Areas · doi:10.5281/zenodo.8211022 · Cited by 1
Мясоедова, М. А., & Мясоедова, З. П. (2020) · Sovremennye informacionnye tehnologii i IT-obrazovanie. · doi:10.25559/sitito.16.202002.379-388 · Cited by 1
Learning spatiotemporal lip dynamics in 3D point cloud stream for visual voice activity detection
Zhang, J., Cao, J., & Sun, J. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105410 · Cited by 1
Audio-Visual Multi-modal Meeting Recording System
Yang, W., Li, P., Yang, W., Liu, Y., Petrosian, O., & Yin, L. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-43789-2_15 · Cited by 1
Kyoung, M., & Song, H. J. (2023) · IEEE Access · doi:10.1109/access.2023.3325474 · Cited by 1
Yu, S., & Yang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53308-2_17 · Cited by 1
Audio–video syncing with lip movements using generative deep neural networks
Mathew, A., Saldanha, A., & Babu, C. N. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18695-x · Cited by 1
RETRACTED: Speech recognition in digital videos without audio using convolutional neural networks
Embriz-Islas, C., Benavides-Álvarez, C., Avilés‐Cruz, C., Zúñiga‐López, A., Ferreyra-Ramírez, A., & Rodríguez-Martínez, E. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-219346 · Cited by 1
RetractedZhao, G., Liu, Y., Wang, X., & Yan, F. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658055 · Cited by 1
Multimodal Synchronization Detection: A Transformer-Based Approach Using Deep Metric Learning
Varshney, M., Mukherji, M., Raja, G. S., & Banerjee, K. (2024) · International Conference on Artificial Intelligence for Internet of Things (AIIoT) · doi:10.1109/aiiot58432.2024.10574702 · Cited by 1
Lip and Speech Synchronization using Supervised Contrastive Learning and Cross-Modal Attention
Varshney, M., Mukherji, M., Senthil Raja, G., Ganesh, A., & Banerjee, K. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581985 · Cited by 1
The Whu Wake Word Lipreading System for the 2024 Chat-Scenario Chinese Lipreading Challenge
Wang, H., Li, C., Su, F., Liu, J., Suo, H., & Li, M. (2024) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew63481.2024.10645425 · Cited by 1
Pose Preserving Landmark Guided Neural Radiation Fields for Talking Portrait Synthesis
Xiong, Z., Huang, H., Tan, J., & Yang, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0116-5_36 · Cited by 1
Music Versus Speech Classification Using 3D CNN
Hullur, P., Kulkarni, D., Jainapur, V., Chikkamath, S., Nirmala, S. R., & Budihal, S. V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-97-8861-3_24 · Cited by 1
Chen, H., Wang, C., Du, J., Yang, C.-H. H., & Qi, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890305 · Cited by 1
Qiang, W., Chen, S., Zhang, X., Zhang, Y., Wang, X., Zheng, C., Xie, L., Ye, Y., & Yin, E. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6963-9_19 · Cited by 1
Landmark Guided Visual Feature Extractor for Visual Speech Recognition with Limited Resource
Yang, L., Jin, J., Zhang, M., He, Y., Chen, B., & Wang, S. (2025) · doi:10.1145/3728481.3762165 · Cited by 1
MVIB-Lip: Multi-View Information Bottleneck for Visual Speech Recognition via Time Series Modeling
Li, Y., Sun, H., Cai, J., & Wu, J. H. (2025) · Entropy · doi:10.3390/e27111121 · Cited by 1
Xue, J., Liu, X., Wu, X., Yu, F. R., & Wang, J. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228396 · Cited by 1
Real-time Lipreading: Effects of Compression and Frame-rate
Maan, R. (2019) · Research Repository (Delft University of Technology) · Cited by 1
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.