Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 501–600 of 1,001
Barrington, S., Bohacek, M., & Farid, H. (2024) · arXiv · doi:10.48550/arxiv.2408.05366 · Cited by 9
Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness
Nguyen, L. T. P., Yu, Z., Cao, K. Q. N., Guo, Y., Pham, T. H., Nguyen, T. T., Vo, T. N. D., Poon, L., Nguyen, T. K., Lee, Y. J., et al. (2026) · Interspeech (USB) · doi:10.21437/interspeech.2026-581 · Cited by 9
Sensing technologies for silent speech interfaces
Tang, C., Qi, L., Gao, S., Zhang, Z., Yi, W., Xu, M., Occhipinti, E., Pan, Y., & Occhipinti, L. G. (2026) · Nature Sensors · doi:10.1038/s44460-025-00010-2 · Cited by 9
Towards Robust Speech Recognition Model Using Deep Learning
Kumar, L. A., Renuka, D. K., & Chinnan, S. P. M. (2023) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois56541.2023.10100390 · Cited by 9
SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
Fernandez-Lopez, A., Chen, H., Ma, P., Haliassos, A., Petridis, S., & Pantić, M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-462 · Cited by 9
HNet: A deep learning based hybrid network for speaker dependent visual speech recognition
Chandrabanshi, V., & Domnic, S. (2024) · International Journal of Hybrid Intelligent Systems · doi:10.3233/his-240014 · Cited by 9
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challenge
Chen, C., Liu, Z., Li, X., Li, L., & Wang, D. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2509 · Cited by 9
MILG: Realistic lip-sync video generation with audio-modulated image inpainting
Bao, H., Zhang, X., Wang, Q., Liang, K., Wang, Z., Ji, S., & Chen, W. (2024) · Visual Informatics · doi:10.1016/j.visinf.2024.08.002 · Cited by 9
Robust Audiovisual Speech Recognition Models with Mixture-of-Experts
Wu, Y., Peng, Y., Lu, Y., Chang, X., Song, R., & Watanabe, S. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832196 · Cited by 9
Target speaker lipreading by audio–visual self-distillation pretraining and speaker adaptation
Zhang, J.-X., Mao, T., Guo, L., Li, J., & Zhang, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126741 · Cited by 9
Listening While Speaking and Visualizing: Improving ASR Through Multimodal Chain
Effendi, J., Tjandra, A., Sakti, S., & Nakamura, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003899 · Cited by 9
A Neural Relation Extraction Model for Distant Supervision in Counter-Terrorism Scenario
Hou, J., Li, X., Zhu, R., Zhu, C., Wei, Z., & Zhang, C. (2020) · IEEE Access · doi:10.1109/access.2020.3042672 · Cited by 9
Li, Q., Huang, Q., Zhou, Y., Yang, T., Ran, M., & Liu, X. (2021) · IEEE Access · doi:10.1109/access.2021.3078709 · Cited by 9
Audiovisual speech recognition for Kannada language using feed forward neural network
Shashidhar, R., & Patilkulkarni, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07249-7 · Cited by 9
Lipreading Using Liquid State Machine with STDP-Tuning
Yu, X., Wan, Z., Shi, Z., & Wang, L. (2022) · Applied Sciences · doi:10.3390/app122010484 · Cited by 9
On Robustness to Missing Video for Audiovisual Speech Recognition
Chang, O., Braga, O., Liao, H., Serdyuk, D., & Siohan, O. (2022) · Transactions on machine learning research · doi:10.48550/arxiv.2312.10088 · Cited by 9
Visual Hallucination Elevates Speech Recognition
Zhang, F., Zhu, Y., Wang, X., Chen, H., Sun, X., & Xu, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29926 · Cited by 9
Sterpu, G., & Harte, N. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101349 · Cited by 9
Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition
Yu, W., Zeiler, S., & Kolossa, D. (2022) · Sensors · doi:10.3390/s22155501 · Cited by 9
Audio-Video detection of the active speaker in meetings
Madrigal, F., Lerasle, F., Pibre, L., & Ferrané, I. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412681 · Cited by 9
Multimodal Topic Learning for Video RecommendationPreprint
Pu, S., He, Y., Li, Z., & Zheng, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13373 · Cited by 9
Lip-reading with Hierarchical Pyramidal Convolution and Self-AttentionPreprint
Chen, H., Du, J., Hu, Y. H., Dai, L.-R., Lee, C., & Yin, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14360 · Cited by 9
From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative BootstrappingPreprint
He, X., Zhang, H., Chen, H., Zheng, C., Chen, L., Tang, S., Huang, J., Liu, X., Wan, P., & Wu, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.25066 · Cited by 9
Ríos-Urrego, C. D., Escobar-Grisales, D., & Orozco‐Arroyave, J. R. (2024) · Diagnostics · doi:10.3390/diagnostics15010073 · Cited by 8
Automatic visual lip reading: A comparative review of machine-learning approaches
Rezaee, K., & Yeganeh, M. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107171 · Cited by 8
Zhu, X., Wang, S., Yang, J., Yang, Y., Tu, W., & Wang, Z. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755563 · Cited by 8
Deep Information-Balanced Multimodal Learning
Qin, Y., Feng, Y., Sun, Y., Peng, D., Peng, X., & Hu, P. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3681770 · Cited by 8
Xi, X., & Zhu, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10347-6 · Cited by 8
Data-Driven Advancements in Lip Motion Analysis: A Review
Torrie, S. A., Sumsion, A. W., Lee, D.-J., & Sun, Z. (2023) · Electronics · doi:10.3390/electronics12224698 · Cited by 8
Clarke, J., Gotoh, Y., & Goetze, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389764 · Cited by 8
Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation
Lei, S., Cheng, X., Lyu, M., Hu, J., Tan, J., Liu, R., Xiong, L., Jin, T., Li, X., & Zhao, Z. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.543 · Cited by 8
Self-Supervised Audio-Visual Soundscape Stylization
Li, T., Wang, R., Huang, P.-Y., Owens, A., & Anumanchipalli, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_2 · Cited by 8
CIEASR:Contextual Image-Enhanced Automatic Speech Recognition for Improved Homophone Discrimination
Wang, Z., Rong, Y., Jiang, D. A., Wu, H., Zhou, S., & Xu, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681665 · Cited by 8
PipeOptim: Ensuring Effective 1F1B Schedule With Optimizer-Dependent Weight Prediction
Guan, L., Li, D., Chen, Y., Liang, J., Wang, W., & Lu, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3543225 · Cited by 8
STDNet: Improved lip reading via short-term temporal dependency modeling
Wu, X., Tan, Z., Cheng, Z., & Ru, Y. (2025) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2024.07.003 · Cited by 8
LiMuSE: Lightweight Multi-Modal Speaker Extraction
Liu, Q., Huang, Y., Hao, Y., Xu, J., & Xu, B. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023036 · Cited by 8
Pre-Avatar: An Automatic Presentation Generation Framework Leveraging Talking Avatar
Sun, A., Zhang, X., Ling, T., Wang, J., Cheng, N., & Xiao, J. Z. (2022) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai56018.2022.00153 · Cited by 8
Chen, S., & Li, F. (2023) · IET Blockchain · doi:10.1049/blc2.12056 · Cited by 8
SyncTalklip: Highly Synchronized Lip-Readable Speaker Generation with Multi-Task Learning
Yang, X., Cheng, X., Fu, D., Fang, M., Zuo, J., Ji, S., Zhao, Z., & Jin, T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681386 · Cited by 8
Cogans For Unsupervised Visual Speech Adaptation To New Speakers
Fernandez-Lopez, A., Karaali, A., Harte, N., & Sukno, F. M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053299 · Cited by 8
Randall, N., Šabanović, S., Milojević, S., & Gupta, A. (2021) · International Journal of Social Robotics · doi:10.1007/s12369-021-00776-8 · Cited by 8
Audio-video fusion strategies for active speaker detection in meetings
Pibre, L., Madrigal, F., Equoy, C., Lerasle, F., Pellegrini, T., Pinquier, J., & Ferrané, I. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13746-7 · Cited by 8
Multimodal Learning of Audio-Visual Speech Recognition with Liquid State Machine
Yu, X., Wang, L., Chen, C., Tie, J., & Guo, S. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1645-0_46 · Cited by 8
Dual-Path Cross-Modal Attention for Better Audio-Visual Speech Extraction
Xu, Z., Fan, X., & Hasegawa‐Johnson, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096732 · Cited by 8
Toward Language-independent Lip Reading: A Transfer Learning Approach
Jitaru, A. C., Stefan, L., & Ionescu, B. E. (2021) · International Symposium on Signals, Circuits and Systems · doi:10.1109/isscs52333.2021.9497405 · Cited by 8
Audio-visual speech separation based on joint feature representation with cross-modal attentionPreprint
Xiong, J., Zhang, P., Xie, L., Huang, W., Zha, Y., & Zhang, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.02655 · Cited by 8
Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of VideosPreprint
Waibel, A. H., Behr, M., Eyiokur, F. İ., Yaman, D., Nguyen, T. N., Mullov, C., Demirtaş, M. A., Kantarcı, A., Constantin, S., & Ekenel, H. K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.04523 · Cited by 8
Towards temporal sentence grounding in videos
Zhang, H. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/163788 · Cited by 8
Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning ModelsPreprint
Gu, X., Zeng, W., Zhang, J., Ou, L., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.12082 · Cited by 9
A Unified Framework for Modality-Agnostic Deepfakes DetectionPreprint
Yu, C., Chen, P., Tian, J., Liu, J., Dai, J., Wang, X., Chai, Y., Jia, S., Lyu, S., & Han, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.14491 · Cited by 8
RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech SeparationPreprint
Pegg, S., Li, K., & Hu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.17189 · Cited by 8
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with DiffusionPreprint
Park, S. J., Hong, J. H., Kim, M., & Ro, Y. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05934 · Cited by 8
Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular DataPreprint
Tang, Z., Zhong, Z., He, T., & Friedland, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.16243 · Cited by 8
Deep Learning-Based Lip-Reading for Vocal Impaired Patient Rehabilitation
Innocente, C., Boemio, M., Lorenzetti, G., Pulito, I., Romagnoli, D., Saponaro, V., Marullo, G., Ulrich, L., & Vezzetti, E. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.063186 · Cited by 7
Zhao, J., Jia, Y., Wang, S., Zhou, J., Wang, H., & Qin, Y. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209943 · Cited by 7
DeepFake Detection by Prediction of Mismatch Between Audio and Video Lip Movement
Sornavalli, G., & Vijaybaskar, P. (2024) · International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS) · doi:10.1109/adics58448.2024.10533515 · Cited by 7
Jishnu, T. S., & Antony, A. (2024) · Indian Journal of Data Mining · doi:10.54105/ijdm.a1632.04010524 · Cited by 7
A short-term vehicle speed prediction approach considering dynamic traffic scene
Wang, C., Zhang, W., Wu, C., Ding, H., & Li, Z. (2024) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2024.130182 · Cited by 7
BIAS: A Body-Based Interpretable Active Speaker Approach
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3520030 · Cited by 7
Deep multimodal learning for time series analysis in social computing: a survey
Yang, C., Chen, Y., Li, Z., Wang, X., Shi, K., Yao, L., Xu, G., & Guo, Z. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00363-x · Cited by 7
Modality-Agnostic Deepfakes Detection
Cai, Y., Chen, P., Tian, J., Liu, J., Dai, J., Wang, X., Jia, S., Lyu, S., & Han, J. (2025) · ACM Workshop on Information Hiding and Multimedia Security (IH&MMSec) · doi:10.1145/3733102.3733133 · Cited by 7
Human-Inspired Computing for Robust and Efficient Audio-Visual Speech Recognition
Liu, Q., Wang, J., Wang, Y., Yang, X., Pan, G., & Li, H. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3582069 · Cited by 7
Emotional Speech Recognition Based on Lip-Reading
Рюмина, Е. В., & Ivanko, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20980-2_52 · Cited by 7
Audio-Driven Deformation Flow for Effective Lip Reading
Feng, D., Yang, S., Shan, S., & Chen, X. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956316 · Cited by 7
Rahman, M. M., Tanjim, M. R., Hasan, S. S., Shaiban, S. M., & Khan, M. A. (2022) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3579654.3579677 · Cited by 7
Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video
Kim, M., Kim, C. W., & Ro, Y. M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25998 · Cited by 7
Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model
Hong, J. H., Park, S. J., & Ro, Y. M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.324 · Cited by 7
GLMB 3D Speaker Tracking with Video-Assisted Multi-Channel Audio Optimization Functions
Qian, X., Pan, Z., Zhang, Q., Chen, K., & Lin, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446460 · Cited by 7
Multi-Modality Speech Recognition Driven by Background Visual Scenes
Luo, C., Liu, Y., Sun, W., & Sun, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446142 · Cited by 7
A Review on Generative Adversarial Networks
Yuan, Y., & Guo, Y. (2020) · International Conference on Information Science, Computer Technology and Transportation (ISCTT) · doi:10.1109/isctt51595.2020.00074 · Cited by 7
Zhou, H., Du, J., Chen, H., Jing, Z., Xiong, S., & Lee, C. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-592 · Cited by 7
Learning from Videos with Deep Convolutional LSTM NetworksPreprint
Courtney, L., & Sreenivas, R. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04817 · Cited by 7
Contrastive Learning of Global and Local Audio-Visual Representations.Preprint
Ma, S., Zeng, Z., McDuff, D., & Song, Y. (2021) · arXiv (Cornell University) · Cited by 7
Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video ClassificationPreprint
Zhu, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.04023 · Cited by 7
ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint LearningPreprint
Cheng, X., Wang, Y., Wang, X., Wu, Y., Guan, K., Chen, Y., Zhang, P., Liu, X., Cao, M., & Song, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.24773 · Cited by 7
Zhang, J., Wan, G., Li, J., Gao, J., Zhao, D., & Ling, Z.-H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.22961 · Cited by 7
D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual CaptioningPreprint
Tang, C., Wang, T., Rao, F., Lyu, J., & Zhang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.07960 · Cited by 7
Continuous lipreading based on acoustic temporal alignments
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2024) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-024-00345-7 · Cited by 6
Character-Aware Audio-Visual Subtitling in Context
Huh, J., & Zisserman, A. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_21 · Cited by 6
Self-citationXie, Z., Yang, Y. Z., Yu, Y., Wang, J., Liu, Y., & Jiang, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112884 · Cited by 6
DCIM-AVSR: Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module
Wang, X., Jiang, H., Huang, H., Fang, Y., Xu, M., & Wang, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890272 · Cited by 6
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 6
Enhancing Audiovisual Speech Recognition Through Bifocal Preference Optimization
Wu, Y., Lu, Y., Peng, Y., Wang, X., Song, R., & Watanabe, S. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34741 · Cited by 6
MultiAVSR: Robust Speech Recognition via Supervised Multi-Task Audio–Visual Learning
Torrie, S. A., Wright, K., & Lee, D.-J. (2025) · Electronics · doi:10.3390/electronics14122310 · Cited by 6
Meta-Guided Adaptive Weight Learner for Noisy Correspondence
Mu, C., Yang, E., & Deng, C. D. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730032 · Cited by 6
From Hype to Insight: Rethinking Large Language Model Integration in Visual Speech Recognition
Jain, R., & Harte, N. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464703 · Cited by 6
EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis
Tan, S., Ji, B., & Pan, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3727763 · Cited by 6
A Fast and Lightweight Model for Causal Audio-Visual Speech Separation
Sang, W., Li, K., Yang, R., Huang, J., & Hu, X. (2025) · Frontiers in artificial intelligence and applications · doi:10.3233/faia250845 · Cited by 6
Barrington, S., Boháček, M., & Farid, H. (2026) · doi:10.1109/cvprf71909.2026.00186 · Cited by 6
Li, Y., Qin, X., Liang, C. C., & Wei, M. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8552-4_8 · Cited by 6
Large-Scale Unsupervised Audio Pre-Training for Video-to-Speech Synthesis
Kefalas, T., Panagakis, Y., & Pantić, M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3382500 · Cited by 6
CTC-CNN-Bidirectional LSTM based Lip Reading System
Shilaskar, S., & Iramani, H. (2024) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci59607.2024.10497275 · Cited by 6
AV-CPL: Continuous Pseudo-labeling for Audio-Visual Speech Recognition
Rouditchenko, A., Collobert, R., & Likhomanenko, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93806-1_18 · Cited by 6
Video-to-Audio Generation with Fine-grained Temporal Semantics
Hu, Y., Gu, Y., Li, C., Chen, R., & Yu, D. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1119 · Cited by 6
Audio-visual speech enhancement in noisy environments using emotion-based contextual cues
Hussain, T., Saleem, N., Dashtipour, K. K., Ahmed, S., Hou, J.-C., Anwar, U., Tsao, Y., Arslan, T., & Hussain, A. (2026) · The Journal of the Acoustical Society of America · doi:10.1121/10.0042239 · Cited by 6
Multimodal Corpus Analysis of Autoblog 2020: Lecture Videos in Machine Learning
Hernandez, A., & Yang, S. H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87802-3_24 · Cited by 6
Audio-Visual Speech Recognition using 3D Convolutional Neural Networks
Belhan, C., Fikirdanis, D., Cimen, O., Pasinli, P., Akgun, Z., Yayci, Z. O., & Türkan, M. (2021) · Innovations in Intelligent Systems and Applications Conference (ASYU) · doi:10.1109/asyu52992.2021.9599016 · Cited by 6
A Generative Answer Aggregation Model for Sentence-Level Crowdsourcing Tasks
Wang, S., & Dang, D. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3142821 · Cited by 6
Wavsyncswap: End-To-End Portrait-Customized Audio-Driven Talking Face Generation
Bao, W., Chen, L., Zhou, C., Yang, S., & Wu, Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094807 · Cited by 6
Kim, J.-E., & Kim, H. Y. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.08.009 · Cited by 6
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.