Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 701–800 of 1,001
Audiovisual speech representation learning applied to emotion recognition
Sadok, S. (2024) · theses.fr (ABES) · doi:10.70675/0f15a36fzecf8z4cffz94e7z120e7d8fd39d · Cited by 4
MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generation
Wang, S., Liu, J., Wang, W., Jin, Y., Du, J., & Han, Z. J. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3604109 · Cited by 3
EchoLip: Pushing the Limit of Acoustic-Based Silent Speech Interface on Mobile Devices
Akbar, A. J., Guo, K., Zhang, Q., & Wang, D. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3603582 · Cited by 3
AV-RISE: Hierarchical Cross-Modal Denoising for Learning Robust Audio-Visual Speech Representation
Zhao, Z., Lin, Y., Guo, D., & Fan, J. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755639 · Cited by 3
AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition
Xue, J., Liu, X., Wu, X., Yin, X., Huang, D., & Yu, F. (2025) · doi:10.1145/3728425.3759905 · Cited by 3
Data-driven bayesian-guided activation functions for multi-task pattern recognition
Bai, R.-J., Li, L., Li, Z., Guo, J., Zhao, C., Zeng, W., Feng, H., Wei, H., & Chen, P. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112911 · Cited by 3
Assessing True Generalisability of Audio-Visual Speech Recognisers
Lin, Z., Petridis, S., Pantić, M., & Harte, N. (2026) · Interspeech (USB) · doi:10.21437/interspeech.2026-2583 · Cited by 3
Significance of Convolutional Neural Network in View of Lip Reading for Speech-Impaired People
Sandi, V., & Shoba, S. (2023) · International Conference on Emerging Frontiers in Electrical and Electronic Technologies (ICEFEET) · doi:10.1109/icefeet59656.2023.10452196 · Cited by 3
Cantonese sentence dataset for lip‐reading
Xiao, Y., Liu, X., Teng, L., Zhu, A., Tian, P., & Huang, J. (2024) · IET Image Processing · doi:10.1049/ipr2.13123 · Cited by 3
Ithal, B. N. V., Lagan, T. G., Sudheer, R., Swathi Rupali, N., V, & Mamatha, H. R. (2024) · International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE) · doi:10.1109/amathe61652.2024.10582171 · Cited by 3
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
Fernandez-Lopez, A., Chen, H., Ma, P., Yin, L., Xiao, Q., Petridis, S., Liu, S., & Pantić, M. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-1153 · Cited by 3
Deep Learning Model Efficient Lip Reading by Using Connectionist Temporal Classification Algorithm
Vidhya, B., Ramkumar, M. O., & Yazhini, K. (2024) · Students Conference on Engineering and Systems · doi:10.1109/sces61914.2024.10652291 · Cited by 3
Hard sample semantic reconstruction for mandarin visual speech recognition
Luo, D., Xie, D., Zhang, Y., Xie, W., & Sun, B. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105066 · Cited by 3
Li, Y., Yao, Q., & Xu, X. D. (2025) · Sensors · doi:10.3390/s25072279 · Cited by 3
DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility
Liu, Y., Fang, Y. C., & Lin, Z. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.130 · Cited by 3
CFLip: Generalizing Lipreading to Unseen Speakers by Learning Common Features
Yu, L., Xue, F., Guo, D., Tang, S., Li, P., Li, S., & Hong, R. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3586845 · Cited by 3
Research progress of frontier image processing in medical endoscopes
Wu, J., Yuan, Y., Liu, L., Cui, H., Xu, T., Zhou, M., Chen, Z., & Xu, B. (2023) · Perioperative Precision Medicine · doi:10.61189/663074tcakcn · Cited by 3
Analyzing the effectiveness of post-learning quantization for optimizing neural networks
Tatarnikova, T. M., & Raskopina, A. S. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-4-10 · Cited by 3
AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies
Acosta-Triana, J.-M., Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3hc4ngie8crs · Cited by 3
Phoneme-based lipreading of silent sentences
Deng, M., & Xiong, S. (2022) · Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC) · doi:10.1109/ipec54454.2022.9777317 · Cited by 3
Siamese decoupling network for speaker-independent lipreading
Lu, L., Xu, X., & Fu, J. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.3.033045 · Cited by 3
Cross-modal mask fusion and modality-balanced audio-visual speech recognition
Li, B., Wu, Z., & Wang, Y. (2022) · IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics55264.2022.9873574 · Cited by 3
Continuous Phoneme Recognition based on Audio-Visual Modality Fusion
Richter, J., Liebold, J., & Gerkmann, T. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892053 · Cited by 3
Deep Learning Approach For Human Emotion-Gender-Age Recognition
Suraj, S. J., & M, S. V. (2022) · International Conference on Communication, Computing and Industry 4.0 (C2I4) · doi:10.1109/c2i456876.2022.10051197 · Cited by 3
Russian Language Speech Generation from Facial Video Recordings Using Variational Autoencoder
Leonov, M. M., Soroka, A. A., & Trofimov, A. G. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-44865-2_51 · Cited by 3
Dip Into: A Novel Method for Visual Speech Recognition using Deep Learning
Kuriakose, L. K., Sinciya, P. O., Joseph, M. R., Namita, R., Nabi, S., & Lone, T. A. (2023) · Annual International Conference on Emerging Research Areas and International Conference on Microelectronics, Communications and Renewable Energy · doi:10.1109/aicera/icis59538.2023.10420231 · Cited by 3
Deep hybrid architectures and DenseNet35 in speaker-dependent visual speech recognition
Seegehalli, P. J., & Krupa, B. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03123-2 · Cited by 3
Data Analytics and Artificial Intelligence
Saini, R., Liwicki, M., & Jara-Valera, A. J. (2023) · Springer handbooks · doi:10.1007/978-3-031-39650-2_18 · Cited by 3
Real Time Online Visual End Point Detection Using Unidirectional LSTM
Sharma, T., Aralikatti, R. C., Margam, D. K., Thanda, A., Roy, S., Kandala, P. A., & Venkatesan, S. M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3253 · Cited by 3
Optimised preprocessing for automatic mouth gesture classification
Brumm, M., & Grigat, R. (2020) · doi:10.15480/882.3614 · Cited by 3
Hlaváč, M., Gruber, I., Železný, M., & Karpov, A. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60276-5_18 · Cited by 3
Streaming Audio-Visual Speech Recognition with Alignment Regularization
Ma, P., Moritz, N., Petridis, S., Fuegen, C., & Pantić, M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-487 · Cited by 3
On the Importance of Video Action Recognition for Visual LipreadingPreprint
Weng, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09616 · Cited by 3
Shashidhar, R., & Kulkarni, S. P. (2021) · Research Square · doi:10.21203/rs.3.rs-173380/v1 · Cited by 3
VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice OverPreprint
Lu, J., Şişman, B., Liu, R., Zhang, M., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03342 · Cited by 3
A Unified Compression Framework for Efficient Speech-Driven Talking-Face GenerationPreprint
Kim, B.-K., Kang, J., Seo, D., Park, H., Choi, S., Song, H., Kim, H., & Lim, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.00471 · Cited by 3
Audio-visual video-to-speech synthesis with synthesized input audioPreprint
Kefalas, T., Panagakis, Y., & Pantić, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16584 · Cited by 3
Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-ReadingPreprint
Luo, S., Yang, S., Shan, S., & Chen, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05058 · Cited by 3
Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial CuesPreprint
Wang, F., Yang, S., Shan, S., & Chen, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.14275 · Cited by 3
Artificial Neural Networks Trained on Noisy Speech Exhibit the McGurk EffectPreprint
Grasse, L., & Tata, M. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05715 · Cited by 3
CEM-Net: Cross-Emotion Memory Network for Emotional Talking Face GenerationPreprint
Wu, K., Li, P., Fu, J., Yang, W., Liu, Y., Zhou, S., & Wang, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.12368 · Cited by 3
Non-Intrusive Automatic Speech Recognition Refinement: A SurveyPreprint
Peyghan, M. R., Roudi, S. S., Zouashkiani, S., Amini, S., Rajabi, F., & Ghaemmaghami, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.07285 · Cited by 3
DeepFake Doctor: Diagnosing and Treating Audio-Video Fake DetectionPreprint
Klemt, M., Segna, C., & Rohrbach, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.05851 · Cited by 3
Zhao, J., Jia, Y., Wang, S., Zhou, J., Wang, H., & Qin, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.15066 · Cited by 3
Human-Centric Content Generation with Diffusion Models: A SurveyPreprint
Yang, S., Lyu, Y., Chen, Z., Li, Y., Dong, B., Han, X., Yang, P., Wang, Z., Rao, A., Liu, Z., et al. (2026) · doi:10.36227/techrxiv.177138906.63173113/v1 · Cited by 3
LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognitionPreprint
Kwak, D., Choi, J., Lee, S., & Chung, J. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.27866 · Cited by 3
Self-citationSpeaker Adaptation for Lip Reading with Robust Entropy Minimization and Adaptive Pseudo Labels
Shen, H., Jiang, W., & Huang, J. (2024) · International Conference on Computing, Machine Learning, and Data Science (CMLDS) · doi:10.1145/3661725.3661767 · Cited by 2
MambaTalk: Speech-Driven 3D Facial Animation with Mamba
Zhu, D., Xu, Z., & Yang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2061-6_23 · Cited by 2
Text-guided dynamic mouth motion capturing for person-generic talking face generation
Liu, K., Wei, J., Yuan, R., Chai, R., He, S., Ma, Z., & Yang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113354 · Cited by 3
Multimodal Silent Speech-based Text Entry with Word-initials Conditioned LLM
Su, Z., Fang, S., & Rekimoto, J. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736612 · Cited by 2
Enhancing Lip Dynamic Authenticity: Learning 3D Temporal Representations for Talking Head Synthesis
Li, H., Chen, H., Huang, Y., Chen, T., & Huang, S. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3750048 · Cited by 2
MSSG: Multi-Scale Speaker Graph Network for Active Speaker Detection
Li, G., Yi, J., Wen, Z., Fu, R., Wang, Y., & Tao, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3608949 · Cited by 2
Development of security requirements for critical infrastructures
Дмитриевич, Е. С., Николаевич, П. А., & Львович, П. П. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-11-18 · Cited by 2
RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling
Pham, L.-K., Tran, T. V. T., Pham, M.-T., & Nguyen, V. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1160 · Cited by 2
Lipvis: A Novel Transient Viseme Extraction Framework for Lip Reading
Yuan, H., Zhang, Y., Zhang, X., Xie, L., & Yin, E. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3627133 · Cited by 2
Reinforcement Learning-Based Edge-Assisted Inference With Multimodal Data
Liu, Y., Xiao, L., Wang, C., Lv, Z., Zhan, Y., Xiao, Y., & Yang, H. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3627276 · Cited by 2
Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework
Yaman, D., Eyiokur, F. İ., Ekenel, H. K., & Waibel, A. H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464517 · Cited by 2
Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation
Wang, W., Pan, Z., Li, X., Wang, S., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3640969 · Cited by 2
Enhancing speech translation with real-time lip synchronization
Koreddi, V., Pandey, S., & Pakray, P. (2025) · International Journal of Speech Technology · doi:10.1007/s10772-025-10234-9 · Cited by 2
Yu, C., Kalkhorani, V. A., Xu, B., & Wang, D. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101924 · Cited by 2
Lip Reading Across Languages: A Cross-Modal Framework Leveraging Foundation Models
Țapu, R., & Mocanu, B. (2025) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi66578.2025.11339323 · Cited by 2
Cappellazzo, U., Petridis, S., & Pantić, M. (2026) · Interspeech (USB) · doi:10.21437/interspeech.2026-417 · Cited by 2
Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection
Peng, J., Wang, Y., Li, Y., Kong, L., & Hong, X. (2026) · doi:10.1109/cvprf71909.2026.00639 · Cited by 2
Deep Learning for the Recognition of Skin Cancer
Alzuabidi, I. A., Jamil, L. S., Ahmed, A. A., Fadhil, S. A., Hasan, M. K., & Ibraheem, A. S. (2023) · International Conference on Advanced Computer Applications (ACA) · doi:10.1109/aca57612.2023.10346847 · Cited by 2
Kumar, S. (2024) · Applied Intelligence and Informatics. · doi:10.1007/978-981-97-0914-4_12 · Cited by 2
Dual Representation Learning From Fetal Ultrasound Video and Sonographer Audio
Gridach, M., Alsharid, M., Jiao, J., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635693 · Cited by 2
Conformer-Based Audio Visual Speech Recognition with Taylor Attention
Xiao, Y., Huang, J., Liu, X., & Zhu, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78186-5_20 · Cited by 2
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective
Chen, C., Li, X., Liu, Z., Li, L., & Wang, D. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800480 · Cited by 2
A Lightweight Dual-Path Conformer Network for Speech Separation
Wang, C., Liu, S., & Chen, S. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9674-8_4 · Cited by 2
Audio to Video: Generating a Talking Fake Agent
Çakır, D., & Kasap, Ö. Y. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-66501-2_17 · Cited by 2
Multimodal Chain: Cross-Modal Collaboration Through Listening, Speaking, and Visualizing
Effendi, J., Tjandra, A., Sakti, S., & Nakamura, S. (2021) · IEEE Access · doi:10.1109/access.2021.3077886 · Cited by 2
A Baseline for ViCo Conversational Head Generation Challenge
Liu, M., Zhai, S., Li, Y., Guan, W., & Nie, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3551569 · Cited by 2
Wang, H., Du, J., Zhou, H., Lu, H., & Cao, Y. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980299 · Cited by 2
Audio-visual feature fusion for improved thoracic disease classification
Bhattacharya, M., & Prasanna, P. (2023) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2654571 · Cited by 2
A survey on Audio-Visual Speech Recognition System
Das, A., Patikar, S., & Medhi, K. (2023) · International Conference on Computing and Communication Systems (I3CS) · doi:10.1109/i3cs58314.2023.10127316 · Cited by 2
Setyaningsih, E. R., Handayani, A. N., Irianto, W. S. G., & Kristian, Y. (2023) · Jurnal Edukasi dan Penelitian Informatika (JEPIN) · doi:10.26418/jp.v9i1.63970 · Cited by 2
Kashiwagi, S., Tanaka, K., Feng, Q., & Morishima, S. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-370 · Cited by 2
GestSync: Determining who is speaking without a talking head
Hegde, S. B., & Zisserman, A. (2023) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2310.05304 · Cited by 2
Self-citationAudio-Visual Speech Enhancement with Selective Off-Screen Speech Extraction
Yoshinaga, T., Tanaka, K., & Morishima, S. (2023) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco58844.2023.10290081 · Cited by 2
Audiovisual Moments in Time: A large-scale annotated dataset of audiovisual actions
Joannou, M., Rotshtein, P., & Noppeney, U. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301098 · Cited by 2
AI Lip Reader Detecting Speech Visual Data with Deep Learning
Geetha, C., Rohan Jai, D., Sandheep Krishna, A., & Reddy, S. S. V. P. (2024) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit61985.2024.10627537 · Cited by 2
Spatially and Temporally Optimized Audio‐Driven Talking Face Generation
Dong, B., Ma, B., & Zhang, L. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15228 · Cited by 2
Audio-Visual Wake-up Word Spotting Under Noisy and Multi-person Scenarios
Li, C., Su, F., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-80136-5_12 · Cited by 2
Shen, S.-F., Guo, M., Tan, J., Duan, S., & Wang, L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126205 · Cited by 2
Toward Micro Lip Reading in the Wild: Dataset, Theory, and Practices
Wang, R., Cheng, Y., Cao, A., Zhong, S., Liao, X., He, Y., Ye, H., Sun, C., Zhao, Q., Du, M., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3539220 · Cited by 2
Realistic Lip Synchronization Using Deep Learning Techniques for Speech Generation
Kavitha, R., K, D. B. P., & Yashwanth, J. (2024) · International BIT Conference (BITCON) · doi:10.1109/bitcon63716.2024.10984909 · Cited by 2
A Study on Speech Assessment with Visual Cues
Ahmed, S., Zezario, R. E., Saleem, N., Hussain, A., Wang, H., & Tsao, Y. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1992 · Cited by 2
The NJU-AALAB Systems for the CHiME-9 MCoRec Challenge
Song, Z., Wang, Y., & Lü, J. (2026) · doi:10.21437/chime.2026-6 · Cited by 2
“Is This an Example Image?” – Predicting the Relative Abstractness Level of Image and Text
Otto, C., Holzki, S., & Ewerth, R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15712-8_46 · Cited by 2
Learning Disentangled Representation in Latent Stochastic Models: A Case Study with Image Captioning
Vyas, N., Rallabandi, S. K., Morishetti, L., Hovy, E. H., & Black, A. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683370 · Cited by 2
A System of Associated Intelligent Integration for Human State Estimation
Matsufuji, A., Hsieh, W.-F., Sato-Shimokawara, E., & Yamaguchi, T. (2019) · Journal of Mechanics Engineering and Automation · doi:10.17265/2159-5275/2019.03.003 · Cited by 2
A novel weight initialization with adaptive hyper-parameters for deep semantic segmentation
Haq, N. U., Khan, A., Rehman, Z. U., Din, A., Shao, L., & Shah, S. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10510-1 · Cited by 2
Resource-efficient TDNN Architectures for Audio-visual Speech Recognition
Koumparoulis, A., Potamianos, G., Thomas, S., & Morais, E. D. S. (2021) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco54536.2021.9616215 · Cited by 2
Arbitrary Voice Conversion via Adversarial Learning and Cycle Consistency Loss
Lian, J., Lin, P., Dai, Y., & Li, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13829-4_49 · Cited by 2
Lip Reading Framework using Deep Learning and Machine Learning
Gianey, H. K., Khandelwal, P., Goel, P., Maheshwari, R., Galhotra, B., & Singh, D. P. (2022) · Wiley eBooks · doi:10.1002/9781119792826.ch4 · Cited by 2
Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2022) · IberSPEECH · doi:10.21437/iberspeech.2022-9 · Cited by 2
Vision-Guided Speaker Embedding Based Speech Separation
Deng, Y., & Wei, Y. (2022) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei56279.2022.9980110 · Cited by 2
Audio-Visual Multi-person Keyword Spotting via Hybrid Fusion
Su, Y., Miao, Z., & Liu, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20500-2_27 · Cited by 2
High-Quality Talking Face Generation via Cross-Attention Transformer
Hou, R., & Zhao, X. F. (2024) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar61438.2024.10671370 · Cited by 2
End-to-End Trainable Self-Attentive Shallow Network for Text-Independent Speaker VerificationPreprint
Park, H., Park, J., & Lee, S. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06146 · Cited by 2
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.