Combining Residual Networks with LSTMs for Lipreading
Open accessAbstract
We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Torch7: A Matlab-like Environment for Machine Learning
Collobert, R., Kavukcuoglu, K., & Farabet, C. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,241 · see citing works
An audio-visual corpus for speech perception and automatic speech recognition
Cooke, M. P., Barker, J., Cunningham, S. P., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,146 · see citing works
Recent advances in the automatic recognition of audiovisual speech
Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817150 · Cited by 752 · see citing works
Audio-visual speech recognition using deep learning
Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., & Ogata, T. (2014) · Applied Intelligence · doi:10.1007/s10489-014-0629-7 · Cited by 581 · see citing works
LipNet: End-to-End Sentence-level LipreadingPreprint
Assael, Y., Shillingford, B., Whiteson, S., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01599 · Cited by 573 · see citing works
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · see citing works
Confusions Among Visually Perceived Consonants
Fisher, C. G. (1968) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1104.796 · Cited by 339 · see citing works
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304 · see citing works
CUAVE: A new audio-visual database for multimodal human-computer interface research
Patterson, E. K., Gurbuz, S., TÜFEKCİ, Z., & Gowdy, J. N. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745028 · Cited by 296 · see citing works
Audio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. E. D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 185 · see citing works
Chiou, G. I., & Hwang, J. (1997) · IEEE Transactions on Image Processing · doi:10.1109/83.605417 · Cited by 139 · see citing works
Word embeddings for speech recognition
Samy Bengio, A., & Heigold, G. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-273 · Cited by 133 · see citing works
Direct Acoustics-to-Word Models for English Conversational Speech Recognition
Audhkhasi, K., Ramabhadran, B., Saon, G., Picheny, M., & Nahamoo, D. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-546 · Cited by 116 · see citing works
Deep complementary bottleneck features for visual speech recognition
Petridis, S., & Pantić, M. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472088 · Cited by 114 · see citing works
Papandreou, G., Katsamanis, A., Pitsikalis, V., & Maragos, P. A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2011515 · Cited by 101 · see citing works
Bulat, A., & Tzimiropoulos, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48881-3_43 · Cited by 74 · see citing works
Self-citationConvolutional aggregation of local evidence for large pose face alignment
Bulat, A., & Tzimiropoulos, Y. (2016) · British Machine Vision Conference (BMVC) · doi:10.5244/c.30.86 · Cited by 74 · see citing works
Improved speaker independent lip reading using speaker adaptive training and deep neural networks
Almajai, I., Cox, S. J., Harvey, R., & Lan, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472172 · Cited by 73 · see citing works
Decoding visemes: Improving machine lip-reading
Bear, H. L., & Harvey, R. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472029 · Cited by 44 · see citing works
Lipreading with Long Short-Term MemoryPreprint
Wand, M., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.08188 · Cited by 36 · see citing works
Improving Lip-reading Performance for Robust Audiovisual Speech Recognition using DNNs
Thangthai, K., Harvey, R., Cox, S., & Theobald, B.-J. (2015) · UEA Digital Repository (University of East Anglia) · Cited by 35 · see citing works
Flagged first, then by citations
1 of the 28 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · see citing works
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416 · see citing works
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation
Michelsanti, D., Tan, Z., Zhang, S.-X., Xu, Y., Yu, M., Yu, D., & Jensen, J. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3066303 · Cited by 326 · see citing works
MMTM: Multimodal Transfer Module for CNN Fusion
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01330 · Cited by 323 · see citing works
End-to-End Audiovisual Speech Recognition
Petridis, S., Stafylakis, T., Ma, P., Cai, F., Tzimiropoulos, G., & Pantić, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461326 · Cited by 301 · see citing works
Self-citationLipreading Using Temporal Convolutional Networks
Martínez, B., Ma, P., Petridis, S., & Pantić, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053841 · Cited by 293 · see citing works
End-To-End Audio-Visual Speech Recognition with Conformers
Ma, P., Petridis, S., & Pantić, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414567 · Cited by 240 · see citing works
Deep Audio-visual Learning: A Survey
Zhu, H., Luo, M., Wang, R., Zheng, A., & He, R. (2021) · Machine Intelligence Research · doi:10.1007/s11633-021-1293-0 · Cited by 185 · see citing works
LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
Yang, S., Zhang, Y., Feng, D., Yang, M., Wang, C., Xiao, J., Long, K., Shan, S., & Chen, X. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756582 · Cited by 175 · see citing works
Large-Scale Visual Speech Recognition
Shillingford, B., Assael, Y., Hoffman, M. W., Paine, T., Hughes, C., Prabhu, U., Liao, H., Sak, H., Rao, K., Bennett, L., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1669 · Cited by 164 · see citing works
Audio-Visual Speech Recognition with a Hybrid CTC/Attention Architecture
Petridis, S., Stafylakis, T., Ma, P., Tzimiropoulos, G., & Pantić, M. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639643 · Cited by 151 · see citing works
Self-citationAuto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., & Pantić, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096889 · Cited by 148 · see citing works
Survey on automatic lip-reading in the era of deep learning
Fernandez-Lopez, A., & Sukno, F. M. (2018) · Image and Vision Computing · doi:10.1016/j.imavis.2018.07.002 · Cited by 131 · see citing works
LCANet: End-to-End Lipreading with Cascaded Attention-CTC
Xu, K., Li, D., Cassimatis, N., & Wang, X. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00088 · Cited by 127 · see citing works
Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition
Song, Q., Sun, B., & Shutao Li, A. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3163771 · Cited by 127 · see citing works
HeadGAN: One-shot Neural Head Synthesis and Editing
Doukas, M. C., Zafeiriou, S., & Sharmanska, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01413 · Cited by 117 · see citing works
Time Domain Audio Visual Speech Separation
Wu, J., Xu, Y., Zhang, S., Chen, L., Yu, M., Xie, L., & Yu, D. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003983 · Cited by 116 · see citing works
Towards Practical Lipreading with Distilled and Efficient Models
Ma, P., Martínez, B., Petridis, S., & Pantić, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415063 · Cited by 115 · see citing works
Sub-word Level Lip Reading With Visual Attention
Prajwal, K. R., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00510 · Cited by 115 · see citing works
Discriminative Multi-Modality Speech Recognition
Xu, B., Lu, C., Guo, Y., & Wang, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01444 · Cited by 112 · see citing works
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116 · see citing works
Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading
Zhang, X., Cheng, F., & Wang, S. (2019) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2019.00080 · Cited by 88 · see citing works
Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers
Zhao, Y., Xu, R., Wang, X., Hou, P., Tang, H., & Song, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6174 · Cited by 97 · see citing works
Improved Speech Reconstruction from Silent Video
Ephrat, A., Halperin, T., & Peleg, S. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.61 · Cited by 96 · see citing works
End-to-End Audiovisual Speech Recognition System With Multitask Learning
Tao, F., & Busso, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2975922 · Cited by 95 · see citing works
Learning to lip read words by watching videos
Chung, J. S., & Zisserman, A. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.02.001 · Cited by 94 · see citing works
Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition
Zhang, Y., Yang, S., Xiao, J., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00134 · Cited by 85 · see citing works
Lip-reading with Densely Connected Temporal Convolutional Networks
Ma, P., Wang, Y., Shen, J., Petridis, S., & Pantić, M. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00290 · Cited by 83 · see citing works
Mutual Information Maximization for Effective Lip Reading
Zhao, X. Y., Yang, S., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00133 · Cited by 81 · see citing works
Deep Learning-Based Automated Lip-Reading: A Survey
Fenghour, S., Chen, D., Guo, K., Li, B., & Xiao, P. (2021) · IEEE Access · doi:10.1109/access.2021.3107946 · Cited by 81 · see citing works
Lip Reading Sentences Using Deep Learning With Only Visual Cues
Fenghour, S., Chen, D., Guo, K., & Xiao, P. (2020) · IEEE Access · doi:10.1109/access.2020.3040906 · Cited by 76 · see citing works
Deformation Flow Based Two-Stream Network for Lip Reading
Xiao, J., Yang, S., Zhang, Y., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00132 · Cited by 75 · see citing works
Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243014 · Cited by 74 · see citing works
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Ren, S., Du, Y., Lv, J., Han, G., & He, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01312 · Cited by 73 · see citing works
Training Strategies for Improved Lip-Reading
Ma, P., Wang, Y., Petridis, S., Shen, J., & Pantić, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746706 · Cited by 71 · see citing works
Combining audio and visual speech recognition using LSTM and deep convolutional neural network
Shashidhar, R., Patilkulkarni, S., & Puneeth, S. B. (2022) · International Journal of Information Technology · doi:10.1007/s41870-022-00907-y · Cited by 66 · see citing works
Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMs
Stafylakis, T., Khan, M. H., & Tzimiropoulos, G. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.10.003 · Cited by 66 · see citing works
Self-citationDeep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64 · see citing works
Gating Neural Network for Large Vocabulary Audiovisual Speech Recognition
Tao, F., & Busso, C. (2018) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2018.2815268 · Cited by 64 · see citing works
Audio-Visual Deep Neural Network for Robust Person Verification
Qian, Y., Chen, Z., & Wang, S. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3057230 · Cited by 63 · see citing works
Visual-Only Recognition of Normal, Whispered and Silent Speech
Petridis, S., Shen, J., Cetin, D., & Pantić, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461596 · Cited by 56 · see citing works
Lip reading with Hahn Convolutional Neural Networks
Mesbah, A., Berrahou, A., Hammouchi, H., Berbia, H., Qjidaa, H., & Daoudi, M. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.04.010 · Cited by 56 · see citing works
A Survey of Research on Lipreading Technology
Hao, M., Mamut, M., Yadikar, N., Aysa, A., & Ubul, K. (2020) · IEEE Access · doi:10.1109/access.2020.3036865 · Cited by 55 · see citing works
USEV: Universal Speaker Extraction With Visual Cue
Pan, Z., Ge, M., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3205759 · Cited by 54 · see citing works
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
Kim, M., Hong, J. H., Park, S. J., & Ro, Y. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00036 · Cited by 55 · see citing works
The Conversation: Deep Audio-Visual Speech Enhancement
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1400 · Cited by 54 · see citing works
Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for LipreadingPreprint
Weng, X., & Kitani, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02540 · Cited by 53 · see citing works
Acceptability of Speech and Silent Speech Input Methods in Private and Public
Pandey, L., Hasan, K., & Arif, A. S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445430 · Cited by 53 · see citing works
Lipreading with DenseNet and resBi-LSTM
Chen, X.-J., Du, J., & Zhang, H.-B. (2020) · Signal, Image and Video Processing · doi:10.1007/s11760-019-01630-1 · Cited by 50 · see citing works
End-to-End Video-to-Speech Synthesis Using Generative Adversarial Networks
Mira, R., Vougioukas, K., Ma, P., Petridis, S., Schuller, B. W., & Pantić, M. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3162495 · Cited by 51 · see citing works
Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
Luo, M., Yang, S., Shan, S., & Chen, X. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00010 · Cited by 50 · see citing works
Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models
Özcan, T., & Baştürk, A. (2019) · Balkan Journal of Electrical and Computer Engineering · doi:10.17694/bajece.479891 · Cited by 49 · see citing works
Video-Driven Speech Reconstruction Using Generative Adversarial Networks
Vougioukas, K., Ma, P., Petridis, S., & Pantić, M. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1445 · Cited by 48 · see citing works
Multi-Grained Spatio-temporal Modeling for Lip-readingPreprint
Wang, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11618 · Cited by 48 · see citing works
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition
Pan, X., Chen, P., Gong, Y., Zhou, H., Wang, X., & Lin, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.308 · Cited by 50 · see citing works
Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection
Min, K., Roy, S., Tripathi, S., Guha, T., & Majumdar, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_22 · Cited by 46 · see citing works
End-to-end visual speech recognition for small-scale datasets
Petridis, S., Wang, Y., Ma, P., Li, Z., & Pantić, M. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.01.022 · Cited by 46 · see citing works
Tsourounis, D., Kastaniotis, D., Theoharatos, C., Kazantzidis, A., & Economou, G. (2022) · Journal of Imaging · doi:10.3390/jimaging8100256 · Cited by 45 · see citing works
LiRA: Learning Visual Speech Representations from Audio Through Self-Supervision
Ma, P., Mira, R., Petridis, S., Schuller, B. W., & Pantić, M. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1360 · Cited by 50 · see citing works
LipType: A Silent Speech Recognizer Augmented with an Independent Repair Model
Pandey, L., & Arif, A. S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445565 · Cited by 41 · see citing works
Selective Listening by Synchronizing Speech With Lips
Pan, Z., Tao, R., Xu, C., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153258 · Cited by 43 · see citing works
Read and Attend: Temporal Localisation in Sign Language Videos
Varol, G., Momeni, L., Albanie, S., Afouras, T., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01658 · Cited by 39 · see citing works
LipFormer: Learning to Lipread Unseen Speakers Based on Visual-Landmark Transformers
Xue, F., Yu, L., Liu, D., Xie, Y., Wu, L. Y., & Hong, R. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3282224 · Cited by 37 · see citing works
Zero-Shot Keyword Spotting for Visual Speech Recognition In-the-wild
Stafylakis, T., & Tzimiropoulos, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_32 · Cited by 37 · see citing works
Self-citationHow to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
Sterpu, G., Saam, C., & Harte, N. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2980436 · Cited by 37 · see citing works
A Lip Reading Model Using CNN with Batch Normalization
NadeemHashmi, S., Gupta, H., Mittal, D., Kumar, K., Nanda, A., & Gupta, S. (2018) · International Conference on Contemporary Computing · doi:10.1109/ic3.2018.8530509 · Cited by 36 · see citing works
Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading
Sheng, C., Zhu, X., Xu, H., Pietikäinen, M., & Liu, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3102433 · Cited by 36 · see citing works
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
Qu, L., Weber, C., & Wermter, S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3191677 · Cited by 36 · see citing works
Deep Audio-Visual Speech Separation with Attention Mechanism
Li, C., & Qian, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054180 · Cited by 33 · see citing works
CroMM-VSR: Cross-Modal Memory Augmented Visual Speech Recognition
Kim, M., Hong, J. H., Park, S. J., & Ro, Y. M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3115626 · Cited by 35 · see citing works
Neural Spatio-Temporal Beamformer for Target Speech Separation
Xu, Y., Yu, M., Zhang, S.-X., Chen, L., Weng, C., Liu, J., & Yu, D. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1458 · Cited by 34 · see citing works
Accurate and Resource-Efficient Lipreading with Efficientnetv2 and Transformers
Koumparoulis, A., & Potamianos, G. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747729 · Cited by 33 · see citing works
A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition
Ivanko, D., Ryumin, D., & Karpov, A. A. (2023) · Mathematics · doi:10.3390/math11122665 · Cited by 35 · see citing works
Vocoder-Based Speech Synthesis from Silent Videos
Michelsanti, D., Slizovskaia, O., Haro, G., Gómez, E., Tan, Z., & Jensen, J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1026 · Cited by 33 · see citing works
Lip reading of words with lip segmentation and deep learning
Miled, M., Messaoud, M. A. B., & Bouzid, A. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13321-0 · Cited by 32 · see citing works
Visual Speech Recognition in a Driver Assistance System
Ivanko, D., Ryumin, D., Kashevnik, A., Axyonov, A., & Karnov, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909819 · Cited by 32 · see citing works
Kim, M., Yeo, J. H., Choi, J., & Ro, Y. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01409 · Cited by 33 · see citing works
Listen, Watch and Understand at the Cocktail Party: Audio-Visual-Contextual Speech Separation
Li, C., & Qian, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2028 · Cited by 30 · see citing works
HLR-Net: A Hybrid Lip-Reading Model Based on Deep Convolutional Neural Networks
Sarhan, A. M., Elshennawy, N. M., & Ibrahim, D. M. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2021.016509 · Cited by 29 · see citing works
Turkish lip-reading using Bi-LSTM and deep learning models
Atila, Ü., & Sabaz, F. (2022) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2022.101206 · Cited by 29 · see citing works
Towards Pose-Invariant Lip-Reading
Cheng, S., Ma, P., Tzimiropoulos, G., Petridis, S., Bulat, A., Shen, J., & Pantić, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054384 · Cited by 28 · see citing works
Self-citationChen, W., Tan, X., Xia, Y., Qin, T., Wang, Y., & Liu, T.-Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413623 · Cited by 28 · see citing works
Deep Word Embeddings for Visual Speech Recognition
Stafylakis, T., & Tzimiropoulos, G. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461347 · Cited by 28 · see citing works
Self-citationVision based Lip Reading System using Deep Learning
Deshmukh, N., Ahire, A., Bhandari, S. H., Mali, A., & Warkari, K. (2021) · International Conference on Computing, Communication and Green Engineering (CCGE) · doi:10.1109/ccge50943.2021.9776430 · Cited by 27 · see citing works
ASR is All You Need: Cross-Modal Distillation for Lip Reading
Afouras, T., Chung, J. S., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054253 · Cited by 25 · see citing works
Ogri, O. E., El-Mekkaoui, J., Benslimane, M., & Hjouji, A. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112430 · Cited by 26 · see citing works
A comparative study of English viseme recognition methods and algorithms
Jachimski, D., Czyżewski, A., & Ciszewski, T. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5217-5 · Cited by 26 · see citing works
Exploring ROI size in deep learning based lipreading
Koumparoulis, A., Potamianos, G., Mroueh, Y., & Rennie, S. J. (2017) · The 14th International Conference on Auditory-Visual Speech Processing · doi:10.21437/avsp.2017-13 · Cited by 21 · see citing works
Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement
Chen, H., Du, J., Hu, Y., Dai, L.-R., Yin, B., & Lee, C. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.06.003 · Cited by 25 · see citing works
Speaker-Adaptive Lip Reading with User-Dependent Padding
Kim, M., Kim, H., & Ro, Y. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_33 · Cited by 25 · see citing works
Prompt Tuning of Deep Neural Networks for Speaker-Adaptive Visual Speech Recognition
Kim, M., Kim, H.-I., & Ro, Y. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3484658 · Cited by 32 · see citing works
Speech Reconstruction With Reminiscent Sound Via Visual Voice Memory
Hong, J. H., Kim, M., Park, S. J., & Ro, Y. M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3126925 · Cited by 24 · see citing works
Power of Attention in MOOC Dropout Prediction
Yin, S., Lei, L., Wang, H., & Chen, W. (2020) · IEEE Access · doi:10.1109/access.2020.3035687 · Cited by 24 · see citing works
Lip Reading by Alternating between Spatiotemporal and Spatial Convolutions
Tsourounis, D., Kastaniotis, D., & Fotopoulos, S. (2021) · Journal of Imaging · doi:10.3390/jimaging7050091 · Cited by 23 · see citing works
Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation
Rahimi, A., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01024 · Cited by 23 · see citing works
Chinese Lip-Reading Research Based on ShuffleNet and CBAM
Fu, Y., Lu, Y., & Ni, R. (2023) · Applied Sciences · doi:10.3390/app13021106 · Cited by 23 · see citing works
Using Lip Reading Recognition to Predict Daily Mandarin Conversation
Haq, M. A., Ruan, S.-J., Cai, W.-J., & Li, L. P. (2022) · IEEE Access · doi:10.1109/access.2022.3175867 · Cited by 22 · see citing works
Efficient DNN Model for Word Lip-Reading
Arakane, T., & Saitoh, T. (2023) · Algorithms · doi:10.3390/a16060269 · Cited by 22 · see citing works
LCSNet: End-to-end Lipreading with Channel-aware Feature Selection
Xue, F., Yang, T., Liu, K., Hong, Z., Cao, M., Guo, D., & Hong, R. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3524620 · Cited by 21 · see citing works
How to Use Time Information Effectively? Combining with Time Shift Module for Lipreading
Hao, M., Mamut, M., Yadikar, N., Aysa, A., & Ubul, K. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414659 · Cited by 21 · see citing works
- Authors
- Stafylakis, T., & Tzimiropoulos, G.
- Affiliations
- University of Nottingham (GB)
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 3652-3656
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2596627958
- Cited by
- 348
- References
- 28
- Citations by year
- 2026: 17 · 2025: 43 · 2024: 35 · 2023: 43 · 2022: 46 · 2021: 65 · 2020: 48 · 2019: 26 · 2018: 22 · 2017: 3
- FWCI
- 19.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Nottingham ePrints (University of Nottingham) (submittedVersion)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- European Commission (HORIZON2020, 706668)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- lip reading, visual speech recognition, ResNet, long short-term memory, spatio-temporal convolution, bidirectional LSTM, word-level recognition, end-to-end architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
