Audio-visual speech recognition using deep learning
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Cited by
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
“achievements have also been obtained in ASR, especially in distant talking situations [132,161,164], audio-visual speech recognition (AVSR) systems [125], and data augmentation on the basis of labelpreserving transformations [27].”
BackgroundA Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning
Dargan, S., Kumar, M., Ayyagari, M. R., & Kumar, G. (2020) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-019-09344-w · Cited by 1,198
A comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801
Application of Deep Learning in Food: A Review
Zhou, L., Zhang, C., Liu, F., Qiu, Z., & He, Y. (2019) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.12492 · Cited by 564
BAT: Deep Learning Methods on Network Intrusion Detection Using NSL-KDD Dataset
Su, T., Sun, H., Zhu, J., Wang, S., & Li, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2972627 · Cited by 434
Local Aggregation for Unsupervised Learning of Visual Embeddings
Zhuang, C., Zhai, A., & Yamins, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00610 · Cited by 417
Deep Learning for Drug-Induced Liver Injury
Xu, Y., Dai, Z., Chen, F., Gao, S., Pei, J., & Lai, L. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00238 · Cited by 382
Deep learning in vision-based static hand gesture recognition
Oyedotun, O. K., & Khashman, A. (2017) · Neural Computing and Applications · doi:10.1007/s00521-016-2294-8 · Cited by 369
Combining Residual Networks with LSTMs for Lipreading
Stafylakis, T., & Tzimiropoulos, G. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-85 · Cited by 348
“More recent works deploy deep learning methods either for extracting ”deep” features ([19] [20] [21]) or for building end-to-end architectures.”
MethodStock market prediction using machine learning classifiers and social media, news
Khan, W., Ghazanfar, M. A., Azam, M. A., Karami, A., Alyoubi, K. H., & Alfakeeh, A. S. (2022) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-01839-w · Cited by 336
Human emotion recognition using deep belief network architecture
Hassan, M. M., Alam, M. G. R., Uddin, M. Z., Huda, S., Almogren, A., & Fortino, G. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.10.009 · Cited by 307
Farsi, B., Amayri, M., Bouguila, N., & Eicker, U. (2021) · IEEE Access · doi:10.1109/access.2021.3060290 · Cited by 292
Yuan, X., Zhou, J., Huang, B., Wang, Y., Yang, C., & Gui, W. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2938890 · Cited by 289
Lipreading Using Temporal Convolutional Networks
Martinez, B., Ma, P., Petridis, S., & Pantic, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053841 · Cited by 276
Lien Minh, D., Sadeghi-Niaraki, A., Huy, H. D., Min, K., & Moon, H. (2018) · IEEE Access · doi:10.1109/access.2018.2868970 · Cited by 272
Comparative study of landslide susceptibility mapping with different recurrent neural networks
Wang, Y., Fang, Z., Wang, M., Peng, L., & Hong, H. (2020) · Computers & Geosciences · doi:10.1016/j.cageo.2020.104445 · Cited by 268
Adeel, A., Gogate, M., & Hussain, A. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.08.008 · Cited by 96
“The multimodal audiovisual speech processing methods have shown significant performance improvement in ASR, speech enhancement and speech separation [21] [22] [23] [24] [25].”
BackgroundExploring ROI size in deep learning based lipreading
Koumparoulis, A., Potamianos, G., Mroueh, Y., & Rennie, S. J. (2017) · The 14th International Conference on Auditory-Visual Speech Processing · doi:10.21437/avsp.2017-13 · Cited by 21
Probabilistic inference of Bayesian neural networks with generalized expectation propagation
Zhao, J., Liu, X., He, S., & Sun, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.060 · Cited by 20
Metadata
- Authors
- Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., & Ogata, T.
- Venue
- Applied Intelligence
- Published
- 2014-12-20
- Volume
- 42
- Issue
- 4
- Pages
- 722-737
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W2076462394
- Cited by
- 579
- References
- 46
- FWCI
- 26.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.