The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Audio-visual speech recognition using deep learning
Open accessAbstract
Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 1×
“One of the most successful applications of DNNs to ASR is the deep neural network hidden Markov model (DNNHMM) [12, 36], which replaces the conventional Gaussian mixture model (GMM) with a DNN to represent the direct projection between HMM states and corresponding acoustic feature inputs.”
Method(2011) · Cited by 2,752
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,115
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · International Conference on Machine Learning · Cited by 1,783
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 3×
“As generally discussed in previous deep learning studies [22, 25], the number and variation of training samples are critical for maximizing the generalization ability of a DNN.”
BackgroundMethodLearning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
GradientBased Learning Applied to Document Recognition
(2009) · Intelligent Signal Processing · doi:10.1109/9780470544976.ch9 · Cited by 1,135
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
(2009) · neural information processing systems · Cited by 962
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288864 · Cited by 876
Building high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 823
Deep learning via Hessian-free optimization
Martens, J. (2010) · International Conference on Machine Learning · Cited by 732
Tandem connectionist feature extraction for conventional HMM systems
Hermansky, H., Ellis, D. P. W., & Sharma, S. (2002) · 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100) · doi:10.1109/icassp.2000.862024 · Cited by 693
(2013) · Cited by 651
Connectionist Speech Recognition
Bourlard, H. A., & Morgan, N. (1994) · Springer eBooks · doi:10.1007/978-1-4615-3210-1 · Cited by 585
Fast Exact Multiplication by the Hessian
Pearlmutter, B. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.1.147 · Cited by 540
Extraction of visual features for lipreading
Matthews, I., Cootes, T. F., Bangham, J. A., Cox, S., & Harvey, R. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982900 · Cited by 537
Using very deep autoencoders for content-based image retrieval.
Krizhevsky, A., & Hinton, G. E. (2011) · The European Symposium on Artificial Neural Networks · Cited by 472
Quantitative association of vocal-tract and facial behavior
Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00048-x · Cited by 420
Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent
Schraudolph, N. N. (2002) · Neural Computation · doi:10.1162/08997660260028683 · Cited by 265
A mew ASR approach based on independent processing and recombination of partial frequency bands
Bourlard, H., & Dupont, S. (2002) · Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96 · doi:10.1109/icslp.1996.607145 · Cited by 248
Connectionist probability estimators in HMM speech recognition
Renals, S., Morgan, N., Bourlard, H., Cohen, M., & Franco, H. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.260359 · Cited by 228
Speechreading by Humans and Machines: Models, Systems, and Applications
(1996) · NATO ASI Series · doi:10.1007/978-3-662-13015-5 · Cited by 226
Palaz, D., Collobert, R., & Magimai-Doss, M. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-438 · Cited by 185
Auto-encoder bottleneck features using deep belief networks
Sainath, T. N., Kingsbury, B., & Ramabhadran, B. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288833 · Cited by 182
Audio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 179
Visual speech recognition using active shape models and hidden Markov models
Luettin, J., Thacker, N. A., & Beet, S. W. (1996) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543246 · Cited by 104
A comparative study of neural network based feature extraction paradigms
Lerner, B., Guterman, H., Aladjem, M. E., & Dinstein, I. (1999) · Pattern Recognition Letters · doi:10.1016/s0167-8655(98)00120-2 · Cited by 86
Improving visual features for lip-reading
Lan, Y.-Y., Theobald, B. J., Harvey, R., & Ong, E.-J. (2010) · UEA Digital Repository (University of East Anglia) · Cited by 74
Abdel‐Hamid, O., & Jiang, H. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-336 · Cited by 53
Dynamic modality weighting for multi-stream hmms inaudio-visual speech recognition
Gurban, M., Thiran, J., Thomas Drugman, A., & Dutoit, T. (2008) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1452392.1452442 · Cited by 48
Feature analysis for automatic speechreading
Scanlon, P., & Reilly, R. B. (2002) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2001.962802 · Cited by 41
Automatic speech recognition improved by two-layered audio-visual integration for robot audition
Yoshida, T., Nakadai, K., & Okuno, H. G. (2009) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2009.5379586 · Cited by 35
Self-citation
Cited by
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
“achievements have also been obtained in ASR, especially in distant talking situations [132,161,164], audio-visual speech recognition (AVSR) systems [125], and data augmentation on the basis of labelpreserving transformations [27].”
BackgroundA Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning
Dargan, S., Kumar, M., Ayyagari, M. R., & Kumar, G. (2020) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-019-09344-w · Cited by 1,198
A comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801
Metadata
- Authors
- Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., & Ogata, T.
- Venue
- Applied Intelligence
- Published
- 2014-12-20
- Volume
- 42
- Issue
- 4
- Pages
- 722-737
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W2076462394
- Cited by
- 579
- References
- 46
- FWCI
- 26.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.