Harnessing GANs for Zero-Shot Learning of New Classes in Visual Speech Recognition
Open accessAbstract
Visual Speech Recognition (VSR) is the process of recognizing or interpreting speech by watching the lip movements of the speaker. Recent machine learning based approaches model VSR as a classification problem; however, the scarcity of training data leads to error-prone systems with very low accuracies in predicting unseen classes. To solve this problem, we present a novel approach to zero-shot learning by generating new classes using Generative Adversarial Networks (GANs), and show how the addition of unseen class samples increases the accuracy of a VSR system by a significant margin of 27% and allows it to handle speaker-independent out-of-vocabulary phrases. We also show that our models are language agnostic and therefore capable of seamlessly generating, using English training data, videos for a new language (Hindi). To the best of our knowledge, this is the first work to show empirical evidence of the use of GANs for generating training samples of unseen classes in the domain of VSR, hence facilitating zero-shot learning. We make the added videos for new classes publicly available along with our code1.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such methods tend to use generative adversarial network models to synthesize and reconstruct visual features of unseen classes, resulting in high accuracy classification and ensuring a balance between seen and unseen class labels.
“Such methods tend to use GAN models to synthesize and reconstruct the visual features of the unseen classes, resulting in high accuracy classification and ensuring a balance between seen and unseen class labels [136, 137].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [137]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
(2017) · Cited by 11,625 · see citing works
Generative Adversarial Networks: An Overview
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765202 · Cited by 4,540 · see citing works
Siamese Neural Networks for One-shot Image Recognition
Koch, G. E., Zemel, R. S., & Salakhutdinov, R. (2015) · Cited by 3,712 · see citing works
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766 · see citing works
(2011) · Cited by 2,752 · see citing works
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509 · see citing works
Low-Shot Visual Recognition by Shrinking and Hallucinating Features
Hariharan, B., & Girshick, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.328 · Cited by 824 · see citing works
Data Augmentation Generative Adversarial NetworksPreprint
Antoniou, A., Storkey, A., & Edwards, H. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04340 · Cited by 797 · see citing works
Thrun, S. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5529-2_8 · Cited by 434 · see citing works
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts
Zhu, Y., Elhoseiny, M., Liu, B., Peng, X., & Elgammal, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00111 · Cited by 412 · see citing works
Lipreading with long short-term memory
Wand, M., Koutník, J., & Schmidhuber, J. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472852 · Cited by 222 · see citing works
OuluVS2: A multi-view audiovisual database for non-rigid mouth motion analysis
Anina, I., Zhou, Z., Zhao, G., & Pietikäinen, M. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163155 · Cited by 147 · see citing works
Link the Head to the "Beak": Zero Shot Learning from Noisy Text Description at Part Precision
Elhoseiny, M., Zhu, Y., Zhang, H., & Elgammal, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.666 · Cited by 123 · see citing works
Continuous Automatic Speech Recognition by Lipreading
Goldschen, A. J., Garcia, O. N., & Petajan, E. D. (1997) · Computational imaging and vision · doi:10.1007/978-94-015-8935-2_14 · Cited by 107 · see citing works
One-Shot Learning with a Hierarchical Nonparametric Bayesian Model
Salakhutdinov, R., Tenenbaum, J. B., & Torralba, A. B. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 100 · see citing works
End-to-End Speech-Driven Facial Animation with Temporal GANsPreprint
Vougioukas, K., Petridis, S., & Pantić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09313 · Cited by 92 · see citing works
Perceptual dominance during lipreading
Easton, R. D., & Basala, M. (1982) · Perception & Psychophysics · doi:10.3758/bf03204211 · Cited by 91 · see citing works
Modality Attention for End-to-end Audio-visual Speech Recognition
Zhou, P., Yang, W., Chen, W., Wang, Y., & Jia, J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683733 · Cited by 77 · see citing works
Multimodal Analysis of User-Generated Multimedia Content
Shah, R. R., & Zimmermann, R. (2017) · Socio-affective computing · doi:10.1007/978-3-319-61807-4 · Cited by 68 · see citing works
Self-citationLow-Shot Learning from Imaginary DataPreprint
Wang, Y.-X., Girshick, R., Hebert, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05401 · Cited by 66 · see citing works
Multi-view Automatic Lip-Reading Using Neural Network
Lee, D.-H., Lee, J., & Kim, K.-E. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54427-4_22 · Cited by 49 · see citing works
Triphone based unit selection for concatenative visual speech synthesis
Huang, F. J., Cosatto, E., & Graf, H. P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745033 · Cited by 32 · see citing works
END-TO-END MULTI-VIEW LIPREADING
Petridis, S., Wang, Y., Li, Z., & Pantić, M. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.161 · Cited by 21 · see citing works
MyLipper: A Personalized System for Speech Reconstruction using Multi-view Visual Feeds
Kumar, Y., Jain, R., Salik, M., et al. (2018) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2018.00-19 · Cited by 20 · see citing works
Self-citationUsing viseme recognition to improve a sign language translation system.
Schmidt, C., Koller, O., Ney, H., et al. (2013) · Open Repository and Bibliography (University of Liège) · Cited by 18 · see citing works
Lipper: Speaker Independent Speech Synthesis Using Multi-View Lipreading
Salik, K. M., Aggarwal, S., Kumar, Y., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.330110023 · Cited by 12 · see citing works
Self-citationHush-Hush Speak: Speech Reconstruction Using Silent Videos
Uttam, S., Kumar, Y., Sahrawat, D., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3269 · Cited by 12 · see citing works
Self-citationMobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices
Shrivastava, N., Saxena, A., Kumar, Y., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3273 · Cited by 8 · see citing works
Self-citationLipper: Synthesizing Thy Speech Using Multi-View Lipreading
Kumar, Y., Jain, R., Salik, K. M., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012588 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 6
- [Amplayo et al. 2018] Amplayo, R. K.; Kim, J.; Sung, S.; and Hwang, S.-w. 2018. Cold-start aware user and product attention for sentiment classification. arXiv preprint arXiv:1806.05507 . · arXiv
- [Chung et al. 2017] Chung, J. S.; Senior, A. W.; Vinyals, O.; and Zisserman, A. 2017. Lip reading sentences in the wild. In Proceedings of CVPR .
- [Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems , 2672-2680.
- [Kumar et al. 2018a] Kumar, Y.; Aggarwal, M.; Nawal, P.; Satoh, S.; Shah, R. R.; and Zimmermann, R. 2018a. Harnessing AI for speech reconstruction using multi-view silent video feed. In Proceedings of the ACM Multimedia Conference .
- [Socher et al. 2013] Socher, R.; Ganjoo, M.; Manning, C. D.; and Ng, A. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems , 935-943.
- [Vinyals et al. 2016] Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , 3630-3638.
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
- Authors
- Kumar, Y., Sahrawat, D., Maheshwari, S., Mahata, D., Stent, A., Yin, Y., Ratn Shah, R., & Zimmermann, R.
- Affiliations
- Adobe Systems (United States) (US)
- National University of Singapore (SG)
- Indraprastha Institute of Information Technology Delhi (IN)
- Bloomberg (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 03
- Pages
- 2645-2652
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2986416418
- Cited by
- 1
- References
- 33
- Citations by year
- 2020: 1
- FWCI
- 0.16
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministry of Education, India
- Indraprastha Institute of Information Technology, Delhi
- Science and Engineering Research Board
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- visual speech recognition, zero-shot learning, generative adversarial networks, lip reading, speaker-independent recognition, Hindi speech recognition, synthetic training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
