CoChat
Sign Up
Cited by 702Speech and Audio ProcessingHand Gesture Recognition SystemsSpeech Recognition and Synthesis

Lip reading sentences in the wild

Open access
Chung, Joon Son, Senior, Andrew, Vinyals, Oriol, Zisserman, Andrew·2017·2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2017.367
Open this paper in CoChat — free accountOpen PDF

Abstract

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a `Watch, Listen, Attend and Spell' (WLAS) network that learns to transcribe videos of mouth motion to characters; (2) a curriculum learning strategy to accelerate training and to reduce overfitting; (3) a `Lip Reading Sentences' (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a significant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that if audio is available, then visual information helps to improve speech recognition performance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2017.367

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Numerous methods have been proposed to detect deepfakes.

    “Thus far, there have been numerous methods proposed to detect deepfakes [25–29].”

    From Deep learning for deepfakes creation and detection: A survey (2022) · cites this paper as [28]

    The Introduction (p. 1) and Related works (pp. 1–2) discuss lip reading and audiovisual speech recognition, not deepfake detection. Searches for deepfake detection and detection of manipulated or synthetic face videos found no passage addressing the claim. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 40

  1. [1][1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 , 2016. 5 - · arXiv
  2. [2][2] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas. Lipnet: Sentence-level lipreading. Under submission to ICLR 2017, arXiv:1611.01599v2 , 2016. 2, 8 - · arXiv
  3. [3][3] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. Proc. ICLR , 2015. 1, 3, 11 -
  4. [4][4] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems , pages 1171-1179, 2015. 2, 4 -
  5. [5][5] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211 , 2015. 1, 2, 3, 4, 6 - · arXiv
  6. [6][6] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In Proc. BMVC. , 2014. 2 -
  7. [7][7] J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio. Endto-end continuous speech recognition using attention-based recurrent nn: first results. arXiv preprint arXiv:1412.1602 , 2014. 2 - · arXiv
  8. [8][8] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems , pages 577-585, 2015. 2, 3, 4 -
  9. [9][9] J. S. Chung and A. Zisserman. Lip reading in the wild. In Proc. ACCV , 2016. 1, 2, 5, 6, 8 -
  10. [10][10] J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV , 2016. 5 -
  11. [11][11] M. Cooke, J. Barker, S. Cunningham, and X. Shao. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America , 120(5):2421-2424, 2006. 1, 2, 6, 8 -
  12. [12][12] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proc. CVPR , 2016. 4 -
  13. [13][13] G. Galatas, G. Potamianos, and F. Makedon. Audio-visual speech recognition incorporating facial depth information captured by the kinect. In Signal Processing Conference (EUSIPCO), 2012 Proceedings of the 20th European , pages 2714-2717. IEEE, 2012. 2 -
  14. [14][14] A. Graves, S. Fern´ andez, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pages 369-376. ACM, 2006. 1, 2 -
  15. [15][15] A. Graves and N. Jaitly. Towards end-to-end speech recog- 16. nition with recurrent neural networks. In Proceedings of the 31st International Conference on Machine Learning (ICML14) , pages 1764-1772, 2014. 1 -
  16. [16][16] A. Graves, N. Jaitly, and A.-r. Mohamed. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on , pages 273-278. IEEE, 2013. 4 -
  17. [17][17] H. Hermansky. Perceptual linear predictive (plp) analysis of speech. the Journal of the Acoustical Society of America , 87(4):1738-1752, 1990. 5 -
  18. [18][18] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 11 -
  19. [19][19] V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1867-1874, 2014. 5 -
  20. [20][20] D. E. King. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research , 10:1755-1758, 2009. 5 -
  21. [21][21] O. Koller, H. Ney, and R. Bowden. Deep learning of mouth shapes for sign language. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 85-91, 2015. 1 -
  22. [22][22] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS , pages 1106-1114, 2012. 1 -
  23. [23][23] Y. Lan, R. Harvey, B. Theobald, E.-J. Ong, and R. Bowden. Comparing visual features for lipreading. In International Conference on Auditory-Visual Speech Processing 2009 , pages 102-106, 2009. 8 -
  24. [24][24] D. Larlus, G. Dorko, D. Jurie, and B. Triggs. Pascal visual object classes challenge. In Selected Proceeding of the first PASCAL Challenges Workshop , 2006. To appear. 1 -
  25. [25][25] R. Lienhart. Reliable transition detection in videos: A survey and practitioner's guide. International Journal of Image and Graphics , Aug 2001. 5 -
  26. [26][26] M. Marschark and P. E. Spencer. The Oxford handbook of deaf studies, language, and education , volume 2. Oxford University Press, 2010. 8 -
  27. [27][27] Y. Mroueh, E. Marcheret, and V. Goel. Deep multimodal learning for audio-visual speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2130-2134. IEEE, 2015. 2 -
  28. [28][28] K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata. Lipreading using convolutional neural network. In INTERSPEECH , pages 1149-1153, 2014. 1 -
  29. [29][29] K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata. Audio-visual speech recognition using deep learning. Applied Intelligence , 42(4):722-737, 2015. 2 -
  30. [30][30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics , pages 311-318. Association for Computational Linguistics, 2002. 7 -
  31. [31][31] S. Petridis and M. Pantic. Deep complementary bottleneck features for visual speech recognition. ICASSP , pages 23042308, 2016. 2 -
  32. [32][32] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, S. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and F. Li. Imagenet large scale visual recognition challenge. IJCV , 2015. 1, 2 -
  33. [33][33] R. Sennrich, B. Haddow, and A. Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 , 2015. 4 - · arXiv
  34. [34][34] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015. 1 -
  35. [35][35] I. Sutskever, O. Vinyals, and Q. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems , pages 3104-3112, 2014. 1, 2, 3, 6 -
  36. [36][36] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. CVPR , 2015. 1 -
  37. [37][37] S. Tamura, H. Ninomiya, N. Kitaoka, S. Osuga, Y. Iribe, K. Takeda, and S. Hayamizu. Audio-visual speech recognition using deep bottleneck features and high-performance lipreading. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) , pages 575-582. IEEE, 2015. 2 -
  38. [39][39] M. Wand, J. Koutn, et al. Lipreading with long short-term memory. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 61156119. IEEE, 2016. 2, 8 -
  39. [40][40] J. Yuan and M. Liberman. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America , 123(5):3878, 2008. 5 -
  40. [41][41] Z. Zhou, G. Zhao, X. Hong, and M. Pietik¨ ainen. A review of recent advances in visual speech decoding. Image and vision computing , 32(9):590-605, 2014. 1

Cited by

Sorted by citations

Authors
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A.
ORCID
Zisserman, Andrew
Venue
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2017-07
Pages
3444-3453
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2017.367
OpenAlex
W2551572271
Cited by
702
References
0
Citations by year
2026: 38 · 2025: 99 · 2024: 105 · 2023: 96 · 2022: 81 · 2021: 84 · 2020: 75 · 2019: 67 · 2018: 45 · 2017: 12
FWCI
38.64
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Engineering and Physical Sciences Research Council (EP/M013774/1)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
lip reading, visual speech recognition, open-world recognition, curriculum learning, audio-visual speech recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review