Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
The DOI matches its Crossref record; 68 of the 71 references it cites were found in a registry or an index.
Abstract
Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models (GMMs) to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feed-forward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks (DNNs) that have many hidden layers and are trained using new methods have been shown to outperform GMMs on a variety of speech recognition benchmarks, sometimes by a large margin. This article provides an overview of this progress and represents the shared views of four research groups that have had recent successes in using DNNs for acoustic modeling in speech recognition.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deep Learning has largely alleviated the need for manual feature extraction, achieving state-ofthe-art performance in fields such as computer vision and speech recognition [13,14].”
From EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces (2018) · cites this paper as [13]
In the citing paper
“In particular, neural networks became competitive, solving problems in speech recognition [68], image recognition [69] and language translation [70].”
From The growing ubiquity of algorithms in society: implications, impacts and innovations (2018) · cites this paper as [68]
In the citing paper
“Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition.”
From Deep learning for time series classification: a review (2019) · cites this paper as Hinton et al., 2012
In the citing paper
“The other more direct reason behind the high accuracies of deep CNNs on time series data is its success in other sequential data such as speech recognition (Hinton et al., 2012) and sentence classification (Kim, 2014)”
From Deep learning for time series classification: a review (2019) · cites this paper as Hinton et al., 2012
In the citing paper
“Recent studies have reported that deep learning methods have been successfully applied to many applications areas e.g., image and video classification [11-13], visual tracking [14], speech recognition [15] and natural language processing [16].”
From Medical image retrieval using deep convolutional neural network (2017) · cites this paper as [15]
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Self-citationReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Self-citationA fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Self-citation(2010) · Cited by 14,134
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 3×
“…learning algorithm for a restricted Boltzmann machine (RBM) which consists of a layer of stochastic binary " visible " units that represent binary input data connected to a layer of stochastic binary hidden units that learn to model significant nonindependencies between the visible units [20].”
InfluentialSelf-citation(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011
Self-citationA Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852 · cited 2×
“Learning the standard deviations of a GRBM is problematic for reasons described in [21], so for pretraining using CD 1 , the data are normalized so that each coefficient has zero mean and unit variance, the standard deviations are set to one when computing ( ) v h p ; , and no noise is added to the…”
Self-citationPerceptual linear predictive (PLP) analysis of speech
Hermansky, H. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.399423 · Cited by 2,536
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Self-citationContractive Auto-Encoders: Explicit Invariance During Feature Extraction
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011) · International Conference on Machine Learning · Cited by 1,603
Cepstral analysis technique for automatic speaker verification
Furui, S. (1981) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1981.1163530 · Cited by 1,173
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010) · Neural Computation · doi:10.1162/neco_a_00052 · Cited by 1,078
On optimization methods for deep learning
Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 1,076
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273556 · Cited by 992
(2009) · neural information processing systems · Cited by 962
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288864 · Cited by 876
Self-citationConversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852
Self-citationMaximum mutual information estimation of hidden Markov model parameters for speech recognition
Bahl, L., Brown, P., de Souza, P., & Mercer, R. (2005) · ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1986.1169179 · Cited by 817 · cited 1×
“…the same size, but even with this constraint it was impossible to train all possible combinations of number of hidden layers [1] [2] [3] [4] [5] [6] [7] [8], number of units per layer [512] [1] [024] [2] [048] [3] [072], and number of frames of acoustic data in the input layer [7] [11] [15] [17]…”
Deep learning via Hessian-free optimization
Martens, J. (2010) · International Conference on Machine Learning · Cited by 732
(2011) · Cited by 713
Self-citationTandem connectionist feature extraction for conventional HMM systems
Hermansky, H., Ellis, D. P. W., & Sharma, S. (2002) · 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100) · doi:10.1109/icassp.2000.862024 · Cited by 693
Seide, F., Li, G., Chen, X., & Yu, D. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163899 · Cited by 632
Self-citationAn application of recurrent nets to phone probability estimation
Robinson, A. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279192 · Cited by 441
Digital speech processing, synthesis, and recognition
(1989) · Speech Communication · doi:10.1016/0167-6393(89)90009-5 · Cited by 412
Boosted MMI for model and feature-space discriminative training
Povey, D., Kanevsky, D., Kingsbury, B., Ramabhadran, B., Saon, G., & Visweswariah, K. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518545 · Cited by 341
Self-citationProbabilistic and Bottle-Neck Features for LVCSR of Meetings
Grezl, F., Karafiat, M., Kontar, S., & Cernocky, J. (2007) · 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07 · doi:10.1109/icassp.2007.367023 · Cited by 340
Large scale discriminative training of hidden Markov models for speech recognition
Woodland, P. C., & Povey, D. (2002) · Computer Speech & Language · doi:10.1006/csla.2001.0182 · Cited by 306
Deep Belief Networks using discriminative features for phone recognition
Mohamed, A.-R., Sainath, T. N., Dahl, G., Ramabhadran, B., Hinton, G. E., & Picheny, M. A. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947494 · Cited by 286
Self-citationPhone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 266
Self-citationUnderstanding how Deep Belief Networks perform acoustic modelling
Mohamed, A.-R., Hinton, G., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288863 · Cited by 265
Self-citationApplication of pretrained deep neural networks to large vocabulary speech recognition
Jaitly, N., Nguyen, P., Senior, A., & Vanhoucke, V. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-10 · Cited by 243
Self-citationMaximum likelihood estimation for multivariate mixture observations of markov chains (Corresp.)
Juang, B.-H., Levinson, S., & Sondhi, M. (1986) · IEEE Transactions on Information Theory · doi:10.1109/tit.1986.1057145 · Cited by 235
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Kingsbury, B. (2009) · 2009 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2009.4960445 · Cited by 232
Self-citationDevelopments and directions in speech recognition and understanding, Part 1 [DSP Education]
Baker, J., Deng, L., Glass, J., Khudanpur, S., Lee, C.-H., Morgan, N., & O'Shaughnessy, D. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2009.932166 · Cited by 228
Self-citationA review of large-vocabulary continuous-speech
Young, S. (1996) · IEEE Signal Processing Magazine · doi:10.1109/79.536824 · Cited by 216
Investigation of full-sequence training of deep belief networks for speech recognition
Mohamed, A.-R., Yu, D., & Deng, L. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-304 · Cited by 210
Self-citationScalable stacking and learning for building deep architectures
Deng, L., Yu, D., & Platt, J. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288333 · Cited by 210
Self-citationGlobal optimization of a neural network-hidden Markov model hybrid
Bengio, Y., De Mori, R., Flammia, G., & Kompe, R. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.125866 · Cited by 203
Yu, D., Deng, L., & Dahl, G. E. (2010) · neural information processing systems · Cited by 189
Self-citationDigital Speech Processing, Synthesis, and Recognition
Furui, S. (2018) · CRC Press eBooks · doi:10.1201/9781482270648 · Cited by 188
Auto-encoder bottleneck features using deep belief networks
Sainath, T. N., Kingsbury, B., & Ramabhadran, B. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288833 · Cited by 182
Self-citationDeep convex net: a scalable architecture for speech pattern classification
Deng, L., & Yu, D. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-607 · Cited by 179
Self-citationDeep and Wide: Multiple Layers in Automatic Speech Recognition
Morgan, N. H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2116010 · Cited by 149
Deng, L., & Sun, D. X. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.409839 · Cited by 144
Self-citationDiscriminative learning in sequential pattern recognition
He, X., Li, D., & Chou, W. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.926652 · Cited by 143
Heterogeneous measurements and multiple classifiers for speech recognition
Halberstadt, A. K., & Glass, J. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-522 · Cited by 122
Pushing the envelope - aside [speech recognition
Morgan, N., Zhu, Q., Stolcke, A., Sönmez, K., Sivadas, S., Shinozaki, T., Mari Ostendorf, E., Jain, P., Heřmanský, H., Ellis, D. P. W., et al. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1511826 · Cited by 96
Speech Recognition Using Augmented Conditional Random Fields
Hifny, Y., & Renals, S. J. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2010286 · Cited by 87
Li, D., Yu, D., & Acero, A. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.878265 · Cited by 84
Self-citationSun, J., & Deng, L. (2002) · The Journal of the Acoustical Society of America · doi:10.1121/1.1420380 · Cited by 71
Self-citationYu, D., Siniscalchi, S. M., Deng, L., & Lee, C. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288837 · Cited by 66
Self-citationSparse Multilayer Perceptron for Phoneme Recognition
Sivaram, G. S. V. S., & Heřmanský, H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2129510 · Cited by 65
Exemplar-Based Sparse Representation Features: From TIMIT to LVCSR
Sainath, T. N., Ramabhadran, B., Picheny, M., Nahamoo, D., & Kanevsky, D. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2155060 · Cited by 65
Self-citationComparing multilayer perceptron to Deep Belief Network Tandem features for robust ASR
Vinyals, O., & Ravuri, S. V. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947378 · Cited by 55
Computational Models for Speech Production
Deng, L. (1999) · Springer eBooks · doi:10.1007/978-3-642-60087-6_20 · Cited by 51
Self-citationImproved phone recognition using Bayesian triphone models
Ming, J., & Smith, F. J. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.674454 · Cited by 51
Product of Experts for Statistical Parametric Speech Synthesis
Zen, H., Gales, M., Nankaku, Y., & Tokuda, K. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2165280 · Cited by 50
Hutchinson, B., Deng, L., & Yu, D. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288994 · Cited by 46
Self-citationDeng, L., & Yu, D. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366945 · Cited by 45
Self-citationSWITCHING DYNAMIC SYSTEM MODELS FOR SPEECH ARTICULATION AND ACOUSTICS
Denc, L. (2004) · Cited by 43
Zweig, G., Nguyen, P., Compernolle, D. V., Demuynck, K., Atlas, L., Clark, P., Sell, G., Wang, M., Sha, F., Hynek Hermansky, B., et al. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947490 · Cited by 37
Self-citationSwitching Dynamic System Models for Speech Articulation and Acoustics
Deng, L. (2004) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4419-9017-4_6 · Cited by 35
Self-citationAn exploration of large vocabulary tools for small vocabulary phonetic recognition
Sainath, T. N., Ramabhadran, B., & Picheny, M. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5373263 · Cited by 34
Self-citationBackpropagation training for multilayer conditional random field based phone recognition
Prabhavalkar, R., & Fosler‐Lussier, E. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495222 · Cited by 30
Improved pre-training of Deep Belief Networks using Sparse Encoding Symmetric Machines
Plahl, C., Sainath, T. N., Ramabhadran, B., & Nahamoo, D. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288836 · Cited by 27
Self-citationGlobal optimization of a neural network-hidden Markov model hybrid
Bengio, Y., Mori, R. D., Flammia, G., & Kompe, R. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.155435 · Cited by 16
Research works printed but not linked: 5
- sainath, 2011, Improvements in using deep belief networks for large vocabulary continuous speech recognition, Speech and Language Algorithm Group IBM Yorktown Heights NY Tech Rep UTML TR 2010-003
- rifai, 0, Contractive autoencoders: Explicit invariance during feature extraction, Proc 28th Int Conf Machine Learning, 833
- yu, 2011, Discriminative pretraining of deep neural networks, U S Patent Filing
- mohamed, 0, Deep belief networks for phone recognition, Proc NIPS Workshop Deep Learning for Speech Recognition and Related Applications
- pearl, 1988, Probabilistic Inference in Intelligent Systems Networks of Plausible Inference
Cited by
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Self-citationAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
“Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization.”
BackgroundMethodDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Self-citation
Metadata
- Authors
- Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., & Kingsbury, B.
- Affiliations
- University of Toronto (CA)
- University of Waterloo (CA)
- Microsoft (United States) (US)
- Google (United States) (US)
- IBM (United States) (US)
- Michigan State University (US)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2012-11
- Volume
- 29
- Issue
- 6
- Pages
- 82-97
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2160815625
- Cited by
- 10,260
- References
- 71
- Citations by year
- 2026: 253 · 2025: 526 · 2024: 610 · 2023: 696 · 2022: 837 · 2021: 978 · 2020: 1232 · 2019: 1313 · 2018: 1220 · 2017: 967 · 2016: 712 · 2015: 498 · 2014: 276 · 2013: 128 · 2012: 9
- FWCI
- 284.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- deep neural network, acoustic modeling, speech recognition, hidden Markov models, Gaussian mixture model, feedforward neural networks, posterior probability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.