Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 601–700 of 1,001
RETRACTED ARTICLE: Application of deep learning in Mandarin Chinese lip-reading recognition
Xing, G., Han, L., Zheng, Y., & Zhao, M. (2023) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-023-02283-y · Cited by 6
RetractedGhosh, S., Aggarwal, T., Hoai, M., & Balasubramanian, N. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.141 · Cited by 7
MultiLingualSync: A Novel Method for Generating Lip-Synced Videos in Multiple Languages
Nanditha, G., Datla, K. V., Kevin, G., Nikitha, R., Pallavi, L., & Babu, C. M. (2023) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon58793.2023.10269988 · Cited by 6
An-ni, & Suharjito, S. (2023) · HighTech and Innovation Journal · doi:10.28991/hij-2023-04-02-010 · Cited by 6
Text-to-Feature Diffusion for Audio-Visual Few-Shot Learning
Mercea, O.-B., Hummel, T., Koepke, A. S., & Akata, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54605-1_32 · Cited by 6
Yu, F., Wang, H., Ma, Z., & Zhang, S. S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447487 · Cited by 6
Construction of word level Tibetan Lip Reading Dataset
Gan, Z., Zeng, H., Yang, H., & Zhou, S. (2020) · International Conference on Information Communication and Signal Processing (ICICSP) · doi:10.1109/icicsp50920.2020.9231973 · Cited by 6
OPT: One-shot Pose-Controllable Talking Head Generation
Liu, J., Wang, X., Fu, X., Chai, Y., Cai, Y., Dai, J., & Han, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094598 · Cited by 6
Resource-Adaptive Deep Learning for Visual Speech Recognition
Koumparoulis, A., Potamianos, G., Thomas, S., & Morais, E. D. S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3003 · Cited by 6
Classification of Handwritten Chinese Numbers with Convolutional Neural Networks
Ameri, R., Alameer, A., Ferdowsi, S., Abolghasemi, V., & Nazarpour, K. (2021) · doi:10.1109/ipria53572.2021.9483557 · Cited by 6
Volumetric Occupancy Detection: A Comparative Analysis of Mapping AlgorithmsPreprint
Gomes, M., Oliveira, M., & Santos, V. M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.03089 · Cited by 6
PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network
Liu, Q., Meng, G., Wu, Z., & Li, H. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-889 · Cited by 5
Audio-Visual Speech Recognition for Human-Robot Interaction: a Feasibility Study
Goetzee, S., Mihhailov, K., Laar, R. V. D., Baraka, K., & Hindriks, K. V. (2024) · IEEE RO-MAN · doi:10.1109/ro-man60168.2024.10731139 · Cited by 5
Li, D., Zhao, K., Wang, W., Ma, Y., Peng, B., Zhang, Y., & Dong, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_21 · Cited by 5
Yang, Z., Yeo, Y. H., Jiang, R., Fu, X., Chen, W., Xi, W., & Zhao, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888331 · Cited by 5
A review of generative models for virtual human motion driving
Wang, P., Peng, H., Mao, S., Wang, S., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-023-4284-x · Cited by 5
Probabilistic Multimodal Learning with von Mises-Fisher Distributions
Hu, P., Qin, Y., Gou, Y., Li, Y., Yang, M., & Peng, X. (2025) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2025/600 · Cited by 5
Khan, N., Wu, B., Tan, S., Ishi, C. T., & Nakadai, K. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750765 · Cited by 5
Nie, F., Ni, J., Zhang, J., Zhang, B., Zhang, W. J., & Li, B. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3673104 · Cited by 5
RADIO: Reference-Agnostic Dubbing Video Synthesis
Lee, D., Kim, C., Yu, S., Yoo, J., & Park, G.-M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00412 · Cited by 5
The meta-learning method for the ensemble model based on situational meta-task
Zhang, Z., Zhou, L., Wu, Y., & Wang, N. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1391247 · Cited by 5
MCT-VHD: Multi-modal contrastive transformer for video highlight detection
Jiang, Y., Luo, S., Guo, L., & Zhang, R. (2024) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2024.104162 · Cited by 5
ProDub: Progressive Growing of Facial Dubbing Networks for Enhanced Lip Sync and Fidelity
Liu, K., Yi, X., & Zhao, X. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688276 · Cited by 5
Shelar, J., Ghatole, D., Pachpande, M., Bhandari, D., & Shinde, S. V. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-16-9447-9_11 · Cited by 5
Present and future challenges in therapeutic designing using computational approaches
Prasad, S., Srivastava, A., Singh, N., Singh, H., Saluja, R., Kumar, A., Yadav, N., & Qidwai, T. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-91172-6.00020-0 · Cited by 5
End-to-End multi-talker audio-visual ASR using an active speaker attention module
Rose, R. C., & Siohan, O. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10866 · Cited by 5
Comparing Speaker Adaptation Methods for Visual Speech Recognition for Continuous Spanish
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2023) · Applied Sciences · doi:10.3390/app13116521 · Cited by 5
Multi-layer encoder–decoder time-domain single channel speech separation
Liu, D., Zhang, T., Christensen, M. G., Yi, C., & Wei, Y. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.03.020 · Cited by 5
Sla-former: conformer using shifted linear attention for audio-visual speech recognition
Xiao, Y., Huang, J., Liu, X., & Zhu, A. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01451-x · Cited by 5
Text Extraction and Translation Through Lip Reading using Deep Learning
Putcha, S. T. K., Rajam, Y. S. V., Sugamya, K., & Gopala, S. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01506156 · Cited by 5
Dashtipour, K. K., Gogate, M., Hussain, A., Buck, B., Anwary, A. R., Arslan, T., & Hussain, A. (2024) · doi:10.21437/avsec.2024-16 · Cited by 5
The Sound of Water: Inferring Physical Properties From Pouring Liquids
Bagad, P., Tapaswi, M., Snoek, C. G. M., & Zisserman, A. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3690989 · Cited by 5
Self-citationImproving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation
Goncalves, L., Mathur, P., Niu, X., Lavania, C., Houston, B., Vishnubhotla, S., Sun, L., & Ferritto, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890684 · Cited by 5
Personalized One-Shot Lipreading for an ALS Patient
Sen, B., Agarwal, A., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.401 · Cited by 5
Lohrenz, T., & Fingscheidt, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2560 · Cited by 5
A Survey of Lipreading Methods Based on Deep Learning
HAO, M., MAMUT, M., & Ubul, K. (2020) · International Conference on Image Processing and Machine Vision (IPMV) · doi:10.1145/3421558.3421563 · Cited by 5
Large-vocabulary Audio-visual Speech Recognition in Noisy Environments
Yu, W., Zeiler, S., & Kolossa, D. (2021) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp53017.2021.9733452 · Cited by 5
End-to-End Chinese Lip-Reading Recognition Based on Multi-modal Fusion
Liu, Y., Lin, C., Wang, M., Liang, S., Chen, Z., & Chen, L. (2022) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic57696.2022.10075247 · Cited by 5
Interpretable Multimodal Learning for Intelligent Regulation in Online Payment Systems
Wang, S., & Zhu, D. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/645 · Cited by 5
Decompose the Sounds and Pixels, Recompose the Events
Rao, V. R., Khalil, M. I., Li, H., Dai, P., & Lu, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20111 · Cited by 5
Combined Prediction Method of Short-Term Distance Headway Based on EB-GRA-TCN
Wang, C., Zhang, W., Wu, C., Hu, H., & Zhu, W. (2022) · Journal of Advanced Transportation · doi:10.1155/2022/6456186 · Cited by 5
Short Video-based Advertisements Evaluation System: Self-Organizing Learning ApproachPreprint
Zhang, Y., Tao, F., Liu, X., Su, R., Mei, X., Ding, W., Zhao, Z., Yuan, L., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12662 · Cited by 5
Where and When: Space-Time Attention for Audio-Visual ExplanationsPreprint
Chen, Y., Hummel, T., Koepke, A. S., & Akata, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01517 · Cited by 5
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 5
GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech RecognitionPreprint
Wang, T., Yang, S., Shan, S., & Chen, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.16031 · Cited by 5
Tao, Y., Liu, L., Xu, X., Feng, J., Wang, J., Qin, Y., Mao, S., Liu, W., & (109515), S. W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03672 · Cited by 5
DiVAS: Video and Audio Synchronization with Dynamic Frame Rates
Fernandez-Labrador, C., Akçay, M., Abecassis, E., Massich, J., & Schroers, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02535 · Cited by 4
Ren, W., Hung, K.-H., Chao, R., Li, Y.-J., Wang, H., & Tsao, Y. (2024) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda64382.2024.10800033 · Cited by 4
CTPR: Contrastive transition predictive representation for reinforcement learning
Sun, H., & Wang, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111124 · Cited by 4
Xiao, M., Ye, J., Liu, B., Gao, Z., Xu, K., & Wang, X. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733429 · Cited by 4
Yang, S., Luo, W., Yang, T., Chen, X., Shen, S., Wang, L., Zhao, H., Liu, J., & Huang, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-09046-3 · Cited by 4
Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation
Kim, H. K., & Kim, H. G. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1692 · Cited by 4
Wu, L., Zhang, X., Zhang, Y., Zheng, C., Liu, T., Xie, L., Zheng, C., & Yin, E. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114722 · Cited by 4
ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction
Wu, W., Wang, S., Wu, X., Meng, H. M. L., & Li, H. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3685870 · Cited by 4
Gong, Y., Wu, J. M., Li, Z., Liu, S., Sun, L., & Chen, C. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-16-8048-9_3 · Cited by 4
Cross-language lipreading by reconstructing Spatio-Temporal relations in 3D convolution
Feng, J., & Long, R. (2022) · Displays · doi:10.1016/j.displa.2022.102357 · Cited by 4
Lip Reading Using Temporal Adaptive Module
Huang, J., Teng, L., Xiao, Y., Zhu, A., & Liu, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8141-0_26 · Cited by 4
Multi-Scale Feature Fusion Network for Lip Recognition
Lin, H., Liu, B., Zhang, G., Yin, Q., Yang, L., & Lan, P. (2024) · IEEE International Conference on Power, Electronics and Computer Applications (ICPECA) · doi:10.1109/icpeca60615.2024.10471068 · Cited by 4
He, Y., Seng, K. P., & Ang, L.-M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3432856 · Cited by 4
Yu, B., Liu, D., Shi, H., Chang, G., Wei, J., Sun, L., Tian, S., & Bu, L. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10832087 · Cited by 4
Huu, P. N., & Sy, T. H. (2025) · Modelling and Simulation in Engineering · doi:10.1155/mse/2087573 · Cited by 4
MLLM-Based Speech Recognition: When and How is Multimodality Beneficial?
Guan, Y., Trinh, V. A., Voleti, V., & Whitehill, J. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3721322 · Cited by 4
Liu, Z., Li, X., Guo, L., Li, L., & Wang, D. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249408 · Cited by 4
Pasa, L., Morrone, G., & Badino, L. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054697 · Cited by 4
Augmenting Images for ASR and TTS Through Single-Loop and Dual-Loop Multimodal Chain Framework
Effendi, J., Tjandra, A., Sakti, S., & Nakamura, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2001 · Cited by 4
On‐device audio‐visual multi‐person wake word spotting
Li, Y., Wang, G., Chen, Z., Tang, H., & Liu, H. (2023) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12189 · Cited by 4
Modern automatic recognition technologies for visual communication tools
Yachnaya, V. O., Lutsiv, V. R., & Malashin, R. O. (2023) · Computer Optics · doi:10.18287/2412-6179-co-1154 · Cited by 4
Endangered Tujia language Speech Recognition Research based on Audio-Visual Fusion
Yu, C., Yu, J., Qian, Z., & Tan, Y. (2022) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3582099.3582128 · Cited by 4
Axyonov, A., Рюмина, Е. В., Ryumin, D., Ivanko, D., & Karpov, A. A. (2023) · Scientific and technical journal of information technologies mechanics and optics · doi:10.17586/2226-1494-2023-23-4-767-775 · Cited by 4
RAttSR: A Novel Low-Cost Reconstructed Attention-Based End-to-End Speech Recognizer
Paul, B., & Phadikar, S. (2023) · Circuits Systems and Signal Processing · doi:10.1007/s00034-023-02570-5 · Cited by 4
Face From Voice: PyTorch Adaptation of Speech2Face Framework
Pietkun, K., Gondek, B., & Browarczyk, M. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.381 · Cited by 4
Recognizing Indonesian words based on visual cues of lip movement using deep learning
Rahmatullah, G. M., Ruan, S.-J., & Li, L. P. (2025) · Measurement · doi:10.1016/j.measurement.2025.116968 · Cited by 4
Enhancing visual speech perception through deep automatic lipreading: A systematic review
Rahmatullah, G. M., Ruan, S.-J., Wisesa, I. W. W., & Li, L. P. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110019 · Cited by 4
Fan, X., Gao, H., Chen, Z., Chang, P., Han, M., & Hasegawa‐Johnson, M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00447 · Cited by 4
Wav2Lip Bridges Communication Gap: Automating Lip Sync and Language Translation for Indian Languages
Venkataraghavan, V., Shoba, S., & Kar, A. (2025) · IEEE Access · doi:10.1109/access.2025.3562883 · Cited by 4
Adaptive diffusion landmark dynamic rendering for realistic talking face video generation
Tang, Y., Yazhi, L., Xiong, L., & Li, W. (2025) · The Visual Computer · doi:10.1007/s00371-025-03907-8 · Cited by 4
Enhanced Lip Reading Using Deep Model Feature Fusion: A Study on the MIRACL-VC1 Dataset
Vekkot, S., Gupta, T. S., Karthik, K. P., & Kaushik, D. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.353 · Cited by 4
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
Guan, Y., Trinh, V. A., Voleti, V., & Whitehill, J. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210156 · Cited by 4
Tsunoda, R., Aihara, R., Takashima, R., Takiguchi, T., & Imai, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747204 · Cited by 4
Multimedia Cognition and Evaluation in Open Environments
Feng, W., Li, H., Wang, X. Y., Duan, X., Qian, Z., Liu, W., & Zhu, W. (2023) · International Workshop on Multimedia Content Generation and Evaluation: New Methods and Practice (McGE) · doi:10.1145/3607541.3616823 · Cited by 4
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2f33ifnprupu · Cited by 4
Audio-visual speech processing using deep learning techniques
Ideli, E. (2019) · Summit (Simon Fraser University) · Cited by 4
Visual Speech Recognition in Natural Scenes Based on Spatial Transformer Networks
Yu, J., & Wang, S. (2020) · IEEE International Conference on Anti-counterfeiting, Security, and Identification (ASID) · doi:10.1109/asid50160.2020.9271783 · Cited by 4
End-to-End Audio-Visual Speech Recognition for Overlapping Speech
Rose, R. C., Siohan, O., Tripathi, A., & Braga, O. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1621 · Cited by 4
Audio-Visual Speech Separation with Visual Features Enhanced by Adversarial Training
Zhang, P., Xu, J., Shi, J. Y., Hao, Y., Qin, L., & Xu, B. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533660 · Cited by 4
X-Norm: Exchanging Normalization Parameters for Bimodal Fusion
Yin, Y., Xu, J., Zu, T., & Soleymani, M. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556581 · Cited by 4
Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech
Kyoung, M., Jeon, H., & Park, K. (2023) · IEEE Access · doi:10.1109/access.2023.3254529 · Cited by 4
EvIs-Kitchen: Egocentric Human Activities Recognition with Video and Inertial Sensor Data
Hao, Y., Uto, K., Kanezaki, A., Sato, I., Kawakami, R., & Shinoda, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-27077-2_29 · Cited by 4
Choudhury, A., Roy, P., & Bandyopadhyay, S. (2023) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc56298.2023.10084209 · Cited by 4
Lip-Corrector: Application of BERT-based model in sentence-level lipreading
Zhao, H., Zhang, B., & Yin, Z. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1871/1/012146 · Cited by 4
Bimodal emotion recognition through audio-visual cues
Ghaleb, E. (2021) · Research Publications (Maastricht University) · doi:10.26481/dis.20210708eg · Cited by 4
Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play FrameworkPreprint
Segev, E., Alroy, M., Katsir, R., Wies, N., Shenhav, A., Ben-Oren, Y., Zar, D., Tadmor, O., Bitterman, J., Shashua, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.01715 · Cited by 4
Interpretable Convolutional SyncNetPreprint
Park, S., Yun, J., Lee, D., & Park, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.00971 · Cited by 4
Gesture-Aware Zero-Shot Speech Recognition for Patients with Language DisordersPreprint
Kim, S., Lee, D., Stark, B. C., & Han, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.13983 · Cited by 4
Leveraging Mamba with Full-Face Vision for Audio-Visual Speech EnhancementPreprint
Chao, R., Ren, W., Li, Y.-J., Hung, K.-H., Huang, S.-F., Fu, S., Cheng, W.-H., & Tsao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.13624 · Cited by 4
Li, K. M., Gao, K., & Hu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.23610 · Cited by 4
Visual Acoustic FieldsPreprint
Li, Y., Kim, H., Zhan, F., Qiu, R.-Z., Ji, M., Shan, X., Zou, X., Liang, P. P., Pfister, H., & Wang, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.24270 · Cited by 4
Multi-Sensor and Multi-Modal Localization in Indoor Environments on Robotic Platforms
Sewtz, M. (2025) · elib (German Aerospace Center) · doi:10.5445/ir/1000183842 · Cited by 4
Semantic Satellite Communications for Synchronized Audiovisual ReconstructionPreprint
Liu, F., Jiang, P., Wang, W., Li, X., & Jin, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.10791 · Cited by 4
Deep and statistical learning on audio-visual data for human-machine interface on embedded systems
Pouthier, B. (2024) · theses.fr (ABES) · doi:10.70675/8fc20884z8244z4d31z9a57zc5ed94601377 · Cited by 4
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.