Deep Audio-Visual Speech Recognition
Open accessThe DOI matches its Crossref record, with a small difference in title or year; 70 of the 71 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
Overview
CoChat Summary
## TL;DR
This paper develops Transformer-based audio-visual speech recognition models for open-world sentence-level lip reading and compares CTC vs. sequence-to-sequence training. It also introduces the LRS2-BBC dataset and shows that visual cues improve recognition, especially when audio is noisy.
## Key findings
- On LRS2-BBC lip reading, the best model is TM-seq2seq, which achieves 48.3% WER with a language model, improving by over 22% absolute over the prior 70.4% state of the art.
- On LRS3-TED lip reading, the same video-only TM-seq2seq model sets a baseline of 58.9% WER.
- For audio-visual speech recognition on clean audio, TM-CTC improves from 10.1% WER for audio-only to 8.2% WER for audio-visual; TM-seq2seq also improves from 9.7% to 8.5% on LRS2-BBC.
- With noisy audio, combining audio and video helps substantially: for LRS2-BBC, TM-CTC drops from 53.4% WER (audio-only) to 23.6% (audio-visual), and TM-seq2seq from 57.4% to 34.2%; the paper also notes improvement of up to 30% in WER under loud background noise.
## Methods
- Compared two Transformer-based architectures for character-level speech transcription: TM-seq2seq trained with cross-entropy and TM-CTC trained with CTC loss; both use separate modality encoders and optional external character-level language model decoding.
- Built and evaluated on LRS2-BBC and LRS3-TED, using video of the mouth region sampled at 25 fps and audio features computed as 321-dimensional spectral magnitudes with a 40 ms window and 10 ms hop.
- Training used a four-stage strategy: pretrain the visual front end, generate frozen visual features, train the sequence module, then fine-tune end-to-end; evaluation reports word error rate (WER), with test-time augmentation and beam search decoding.
## Limitations
- The paper states that TM-seq2seq is harder to train and takes longer: the full audio-visual model takes about 8 days to complete the curriculum on a single GeForce Titan X GPU, versus about 5 days for TM-CTC.
- It states that TM-CTC struggles with out-of-sync audio and video compared with TM-seq2seq, even after several epochs of fine-tuning; TM-seq2seq becomes robust after one epoch of calibration.
- Evident from the text: the models are evaluated primarily with WER on the reported datasets, and the dataset is built from BBC television and TED/TEDx sources, so the results may be tied to those domain settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset contains talking faces together with subtitles of what is said.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance even when the audio signal is clean - for example the word error rate is reduced from 10.1% for audio only to 8.2%, when using the audio-visual TM-CTC model.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 71 claims and to verify the 15 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[11] is the first work introducing transformers to VSR.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The experiments clearly showed that the transformer-based backend network performs much better than the RNN-based backend network in the sentence-level VSR task.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [103]
The cited source says what the claim attributes to it.
In the citing paper
“It is built for sentence-level lip reading, a more challenging VSR problem than word-level lip reading.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“LRS2-BBC [11] is a sentence-level audio-visual dataset with a similar data collection pipeline and data source as that LRW dataset.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [11]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Self-citationImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 1
- chung, 2017, Lip reading in profile, Proc Brit Mach Vis Conf
Cited by
Items 901–1,000 of 1,001
Continuous Speech Recognition using EEG and VideoPreprint
Krishna, G., Carnahan, M., Tran, C., & Tewfik, A. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07730 · Cited by 1
Audio-Visual Decision Fusion for WFST-based and seq2seq ModelsPreprint
Aralikatti, R. C., Roy, S., Thanda, A., Margam, D. K., Kandala, P. A., Sharma, T., & Venkatesan, S. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10832 · Cited by 1
Joint Distribution across Representation Space for Out-of-Distribution DetectionPreprint
Xu, J., Zhu, S., Li, Z., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12344 · Cited by 1
Interactive decoding of words from visual speech recognition modelsPreprint
Shillingford, B., Assael, Y., & Denil, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00692 · Cited by 1
Perception Point: Identifying Critical Learning Periods in Speech for Bilingual NetworksPreprint
Saraswat, A., Bhatia, M., Singla, Y. K., Chen, C., & Shah, R. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06507 · Cited by 1
A Closer Look at Audio-Visual Multi-Person Speech Recognition and Active Speaker SelectionPreprint
Braga, O., & Siohan, O. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05684 · Cited by 1
Braga, O., & Siohan, O. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05206 · Cited by 1
Multi-task Learning Approach for Automatic Modulation and Wireless Signal ClassificationPreprint
Jagannath, A., & Jagannath, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10254 · Cited by 1
Audio-visual multi-channel speech separation, dereverberation and recognitionPreprint
Li, G., Yu, J., Deng, J., Liu, X., & Meng, H. M. L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.01977 · Cited by 1
End-to-End Multi-Person Audio/Visual Automatic Speech RecognitionPreprint
Braga, O., Makino, T., Siohan, O., & Liao, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05586 · Cited by 1
A multimodal dynamical variational autoencoder for audiovisual speech representation learningPreprint
Sadok, S., Leglaive, S., Girin, L., Alameda-Pineda, X., & Séguier, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03582 · Cited by 1
OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow GuidancePreprint
Ge, S., Xing, H., Li, Z., & Wu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.14709 · Cited by 1
Out-Of-Distribution Detection for Audio-visual Generalized Zero-Shot Learning: A General FrameworkPreprint
Wen, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.01284 · Cited by 1
Titanic Calling: Low Bandwidth Video Conference from the Titanic WreckPreprint
Eyiokur, F. İ., Huber, C., Nguyen, T. B., Nguyen, T. N., Retkowski, F., Ugan, E. Y., Yaman, D., & Waibel, A. H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.11434 · Cited by 1
Liu, S., Qu, R., & Jin, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.04427 · Cited by 1
Cenacchi, F., Cao, L., McEwan, M., & Richards, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.13802 · Cited by 1
SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery DetectionPreprint
Liang, Y., Yu, M., Li, G., Jiang, J., Li, B., Yu, F., Zhang, N., Meng, X., & Huang, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.09913 · Cited by 1
Mata, D. B. M. D. (2022) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/fgv0-2487 · Cited by 1
Transient Noise Removal via Diffusion-based Speech InpaintingPreprint
Moradi, M., & Gannot, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.08890 · Cited by 1
Real-world listening effort in adult cochlear implant users
Pérez, F. P. (2023) · Cited by 1
From Coarse to Fine: Recursive Audio-Visual Semantic Enhancement for Speech SeparationPreprint
Xue, K., Fan, R., Lixin, Zhao, D., Zhu, C., & Hu, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22425 · Cited by 1
Lip-Siri: Contactless Open-Sentence Silent Speech with Wi-Fi BackscatterPreprint
Tian, Y., Du, H., Gu, C., Zhang, J., Wang, S., Zhou, H., Hou, J., & Li, X.-Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.18177 · Cited by 1
Text-Driven Emotionally Continuous Talking Face GenerationPreprint
Yang, H., Zhao, Y., Zheng, T., Zhang, H., Wang, B., Wu, D., Fu, X., Zhi, X., Huang, Y., & He, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.06071 · Cited by 1
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality PrioritizationPreprint
Wei, S., Luo, C., Zhu, Q., & Luo, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.05773 · Cited by 1
HighSync: High-Quality Lip Synchronization via Latent Diffusion ModelsPreprint
Daghigh, S. F., Mobarakeh, M. I., Alavi, M., & Bagheri, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.16918 · Cited by 1
Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and LipreadingPreprint
del Blanco, E., Gimeno-Gómez, D., Navas, E., Martínez-Hinarejos, C.-D., & Hernáez, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.09667 · Cited by 1
Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings
Kuang, Z., Zong, S., Zhang, J., Chen, J., & Liu, H. (2022) · arXiv · doi:10.48550/arxiv.2210.00434
Effendi, J., Tjandra, A., Sakti, S., & Nakamura, S. (2019) · arXiv (Cornell University)
Roy, A. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.13162
Alexander, J. K. (2019) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)
End-to-End Lip Synchronisation.Preprint
Kim, Y. J., Heo, H. S., Chung, S.-W., & Lee, B. (2020) · arXiv (Cornell University)
Predicting Video features from EEG and Vice versaPreprint
Krishna, G., Tran, C., Carnahan, M., & Tewfik, A. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11235
Axyonov, A., Ivanko, D., Lashkov, I., Ryumin, D., Kashevnik, A. M., & Karpov, A. A. (2020) · Informatization and communication · doi:10.34219/2078-8320-2020-11-5-87-93
End to End Lip Synchronization with a Temporal AutoEncoderPreprint
Shalev, Y., & Wolf, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.16224
Audio-visual deep learning methods for musical instrument classification and separation
Slizovskaia, O. (2020) · Repositori digital de la UPF (Universitat Pompeu Fabra)
Text Recognition Using Silent Speech
Kumar, G. (2021) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2021.37414
Towards reliability-guided information integration in audio-visual speech recognition
Yu, W., Zeiler, S., & Kolossa, D. (2021)
On Off-Screen Speech Separation based on Deep Learning of Audio-Video Fusion
Yu, Y., Liu, Y., & Wei, Y. (2021) · International Conference on Computational Intelligence and Applications (ICCIA) · doi:10.1109/iccia52886.2021.00010
"Syllabic Units Automatically Segmented Data for Continuous Speech Recognition"
Solanki, M. S. (2021) · International Journal of Innovative Research in Computer Science & Technology · doi:10.55524/ijircst.2021.9.6.53
CM-CIF: Cross-Modal for Unaligned Modality Fusion with Continuous Integrate-and-Fire
Jiang, Z., Xu, Y., Xu, Y., Ke, D., & Su, K. (2022) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs55155.2022.9846612
AVATAR submission to the Ego4D AV Transcription ChallengePreprint
Seo, P. H., Nagrani, A., & Schmid, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09966
Synthesizing Talking Face Videos with a Spatial Attention Mechanism
Wang, T., Zhou, C., & Yu, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20233-9_53
Automated dataset collection pipeline for lip motion authentication
Torrie, S. A., Sumsion, A. W., Sun, Z., & Lee, D.-J. (2023) · Electronic Imaging · doi:10.2352/ei.2023.35.5.iriacv-326
Sharma, S., Dhall, A., Kumar, V., & Bawa, V. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.10883
Exploring the Synergy of Multimodal Representations for Robust and Interpretable LearningPreprint
Srivastava, M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3dfbq
Enhance Gesture Recognition via Visual-Audio Modal Embedding
Cao, Y., Fang, Y., & Xiao, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30108-7_33
Lip Detection and Recognition-A Review1
Sharma, S., & Malathi, D. (2023) · International Conference on Networking and Communications (ICNWC) · doi:10.1109/icnwc57852.2023.10127483
A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head GenerationPreprint
Airale, L., Vaufreydaz, D., & Alameda-Pineda, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.03270
A Survey on Talking Head Generation: The Methods, Status and ChallengesPreprint
Cai, Y., Qiao, P., & Li, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4576782
On the Behavior of Audio-Visual Fusion Architectures in Identity Verification TasksPreprint
Claborne, D. M., Slyman, E., & Pazdernik, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05071
Beyond Conversational Discourse: A Framework for Collaborative Dialogue Analysis
Li, Q., Zhang, Z., Liu, Z., Mai, Q., Qiao, W., & Ma, M. (2023) · International Conference on Computer Science and Application Engineering (CSAE) · doi:10.1145/3627915.3628019
Zhen, H., Jiang, C., Zhou, J., Liang, L., & Gao, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8537-1_22
Voice-Driven Virtual Population Type Method Based on Attention Mechanism
RuoFei, C., Runyun, Y., JingCheng, W., & JiChao, L. (2023) · International Conference on Information Communication and Signal Processing (ICICSP) · doi:10.1109/icicsp59554.2023.10390628
Multimodal active speaker detection using cross-attention and contextual information
Mocanu, B., & Țapu, R. (2024) · International Conference on Consumer Electronics · doi:10.1109/icce59016.2024.10444380
Zhang, J., Chen, J., Wang, C., Yu, Z., Qi, T., & Wu, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.18122
Korean Lip-Reading: Data Construction and Sentence-Level Lip-Reading
Cho, S., & Yoon, S. (2024) · Journal of the Korea Institute of Military Science and Technology · doi:10.9766/kimst.2024.27.2.167
Extracting Audio from Image Using Machine Learning
A, M. B. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem31532
Jagannath, A. (2024) · Digital Repository Service (Northeastern University) · doi:10.17760/d20659762
Script Generation for Silent Speech in E-Learning
Hossain, S. M. E., Khairy, S. O. F., Soosaimanickm, A., & Ramadan, R. Α. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-2440-0.ch020
Emotional Neural Textures: Generating Talking-Face Videos with Continuously Controllable EmotionsPreprint
Wan, B., Sun, Z., Lin, M. G., Ye, Z., Wen, Y., & Zhang, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4873630
Lip-synchronized Talking Face Generation with Enhanced Mouth Movement
Liu, F., & Wang, Q. (2024) · International Conference on Computer Vision, Image and Deep Learning (CVIDL) · doi:10.1109/cvidl62147.2024.10603531
Speech Separation Method Based on Visual Modal Scale Fusion
朱, 亚. (2024) · Instrumentation and Equipments · doi:10.12677/iae.2024.123042
Audiovisual Speech Recognition Method Based on Connectionism
Che, N., Zhu, Y., Adetunji, F., Dlamini, K., Shi, L., & Zeng, X. (2024) · Insights of automation in manufacturing. · doi:10.59782/iam.v1i2.225
Schweitzer, M., Montagnini, A., Fourtassi, A., & Schatz, T. (2024) · HAL
Sign Language Transformer using Spatial Representations
Maahir, M., Sheela, K. A., Sathwik, N., & Sandhya, J. (2024) · Engineering World · doi:10.37394/232025.2024.6.21
Édition visuelle pilotée par l'audio à l'aide d'outils de synthèse vocale
Decelle, R., Miguet, S., & Jaillon, T. (2024) · HAL
Lip-reading: Phoneme-to-text Machine Translation using Large Language Models
El-Ghomari, D., El‐Bialy, R., & Chen, D. (2024) · International Conference on Cyber-Physical Social Intelligence (ICCSI) · doi:10.1109/iccsi62669.2024.10799296
Fmfenet: Fast Multiscale Feature Extraction Based on Pyramid Attention Mechanism Network for AsrPreprint
Wang, D., Wan, H., & Majoug, E. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5063627
Speech-Driven Facial Generation
李, 昊. (2025) · Computer Science and Application · doi:10.12677/csa.2025.151020
(2025) · Proceedings of the Institute of Applied Mathematics · doi:10.30546/2225-0530.2024.13.2.026
IMPROVED LIP-READING LANGUAGE USING GATED RECURRENT UNITS
Zulfa, N., Suciati, N., & Hidayati, S. C. (2021) · JUTI Jurnal Ilmiah Teknologi Informasi · doi:10.12962/j24068535.v19i2.a1080
Ismailov, E. (2025) · Problems of Information Society · doi:10.25045/jpis.v16.i1.08
GRS-Wav2lip: General Speech-Driven Lip Synthesis in the Wild
Wu, J., Ye, D., Fu, L., & Zhang, Z. (2024) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc62609.2024.10941899
Lip Reading with Deep Learning: A Comprehensive Analysis of Model ArchitecturesPreprint
Ahmed, A. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3yavr_v1
Specialized Face Encoder for Audio-based Speech to Lip Generation
Li, M., Dai, D., Xue, L., & Tang, P. (2024) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml63543.2024.10957944
CNN LIPNET : Automated Lip Reading Using Deep Convolutional Neural Networks
Ganesh, V., Kumar, O. S. L., Jithendra, P., & Kumar, S. S. (2025) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect63952.2025.10958857
Lip Reading with Deep Learning: A Comprehensive Analysis of Model ArchitecturesPreprint
Cherif, A. (2025) · Research Square · doi:10.21203/rs.3.rs-6502053/v1
Translingo: AI Powered Multilingual Translator
-, V. A. G. R., -, J. J., -, V. S., -, G. V., & -, K. U. V. M. (2025) · International Journal on Science and Technology · doi:10.71097/ijsat.v16.i2.3845
Yusron, A. (2024) · Al-Hijr Journal of Adulearn World · doi:10.55849/alhijr.v3i3.717
Speech-aided facial video super resolution with accurate lip motion and enhanced frequency details
Sharma, S., Singh, V., Dhall, A., & Kumar, V. (2025) · Machine Vision and Applications · doi:10.1007/s00138-025-01699-4
Silent Speech Recognition Using AI and ML
Sailaja, K. S. (2025) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem48292
TransLip: AI-Driven Video Translation and Synchronization
Muralidharan, A., Aji, M., Jacob, R. J., Cherukat, S., & Priya, C. V. (2025) · International Conference on Trends in Engineering Systems and Technologies (ICTEST) · doi:10.1109/ictest64710.2025.11042317
TavNet-An Audiovisual Speech Separation Model Based on Dual-Path Recurrent Neural Networks
Huang, T., Bao, N., Xiang, Q., & Yang, Y. (2025) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice64954.2025.11064412
Masked Appearance Restoration for High Resolution Talking Face Generation
Wen, Y., Cheng, Z., Zhu, C., & Rui, D. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9955-1_39
Formation of a Language Model for Recognizing Spoken Words from the Azerbaijani Lexicon
Rzayev, R. R. O., Ismayilov, E., & Kerimov, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-97992-7_54
Explicit-implicit feature modeling in drama videos: dataset and benchmark
Du, Z., & Wu, X. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06699-5
Advancing Chinese Lip Reading through Contextual Enhancement
Xue, R., Li, J., Zhe-hui, W., & He, W. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099213
Yeni Türkçe Veri Seti ile Evrişimli Sinir Ağları Kullanarak Kelime Seviyesinde Otomatik Dudak Okuma
Berkol, A., Akman, N. P., & Erdem, H. (2025) · International Journal of Multidisciplinary Studies and Innovative Technologies · doi:10.36287/ijmsit.9.1.12
Vietnamese Automatic Speech Recognition Utilizing Audio and Visual Data
Duong, T.-T., Nguyen, V.-M., Phạm, T. T. H., & Le, H.-M. (2025) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr67746.2025.11133884
Preferences and Listening Efficiency of Adults With Cochlear Implants During Online Communication
Pérez, F. P., Hartley, D. E. H., Kitterick, P. T., & Wiggins, I. M. (2025) · Ear and Hearing · doi:10.1097/aud.0000000000001702
Visual Voice: An AI Powered Lip Reading Technology
Kumaran, P., Siddharth, V. V., Bandi, T. R., & Dharavath, N. (2025) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic64892.2025.11158201
Crowd Anomaly Detection Using Deep Learning
Raikar, J., Rajgure, N., Londhe, D., Jadhav, R., Vele, K., & Pujari, A. (2025) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2025.1409000013
Yashina, M. V., Lebedev, A. N., Tatashev, A. G., & Smyk, A. F. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-42-50
Dyomin, A. D., Tereshonok, M. V., & Ilina, O. V. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-57-64
Dataset customization for English words recognition by articulation
Yachnaya, V. O. (2025) · H&ES Research · doi:10.36724/2409-5419-2025-17-2-51-56
Enhancing cross-modal voice-face association with heterogeneous hashing network
Liang, Y., Zhang, H., Liu, X., Yang, X., Wang, F., & Jiang, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01999-9
Shi, H., Qu, L., Liu, Y., Gao, D., Zheng, Y., & Li, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.12761
Multispeaker Audio-Visual Fusion for Intelligent Interaction Systems
Zhang, C., Liu, Y., Liu, Z., Chen, W., & Liu, J. (2025) · IEEE International Conference on Signal Processing and Machine Learning (SPML) · doi:10.1109/spml66318.2025.11199760
Efficient Mouth Alignment for Visual Speech Recognition
Niu, Z., & Mak, B. K.-W. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204267
BALANCING VISUAL DATASET TO ESTIMATE NUMBER OF PHONEMES IN A WORD BY ARTICULATION
Yachnaya, V. O. (2025) · Vestnik komp iuternykh i informatsionnykh tekhnologii · doi:10.14489/vkit.2025.07.pp.015-022
Metadata
- Authors
- Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-12-01
- Volume
- 44
- Issue
- 12
- Pages
- 8717-8727
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2890952074
- Cited by
- 801
- References
- 71
- Citations by year
- 2026: 56 · 2025: 134 · 2024: 155 · 2023: 129 · 2022: 99
- FWCI
- 40.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- EPSRC Programme (Seebibyte EP/M013774/1)
- EPSRC CDT
- Oxford-Google DeepMind Graduate Scholarship
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- audio-visual speech recognition, lip reading, CTC loss, transformer self-attention, audiovisual integration, noisy audio, talking faces
- MeSH terms
- Algorithms, Humans, Lipreading, Speech, Speech Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.