The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition
Abstract
We propose a novel regularized adaptation technique for context dependent deep neural network hidden Markov models (CD-DNN-HMMs). The CD-DNN-HMM has a large output layer and many large hidden layers, each with thousands of neurons. The huge number of parameters in the CD-DNN-HMM makes adaptation a challenging task, esp. when the adaptation set is small. The technique developed in this paper adapts the model conservatively by forcing the senone distribution estimated from the adapted model to be close to that from the unadapted model. This constraint is realized by adding Kullback-Leibler divergence (KLD) regularization to the adaptation criterion. We show that applying this regularization is equivalent to changing the target distribution in the conventional backpropagation algorithm. Experiments on Xbox voice search, short message dictation, and Switchboard and lecture speech transcription tasks demonstrate that the proposed adaptation technique can provide 2%-30% relative error reduction against the already very strong speaker independent CD-DNN-HMM systems using different adaptation sets under both supervised and unsupervised adaptation setups.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 2×
“Recently the context dependent deep neural network hidden Markov models (CD-DNN-HMMs) outperformed the discriminatively trained Gaussian mixture model (GMM) HMMs with 16% [1][2] and 33% [3] relative error reduction, respectively, on the Bing voice search [4] and the Switchboard (SWB) tasks.”
MethodSelf-citationDeep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,115
Self-citationAcoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Conversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852
Self-citationAn investigation of deep neural networks for noise robust speech recognition
Seltzer, M. L., Yu, D., & Wang, Y. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639100 · Cited by 640
Self-citationSeide, F., Li, G., Chen, X., & Yu, D. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163899 · Cited by 632
Self-citationApplication of pretrained deep neural networks to large vocabulary speech recognition
Jaitly, N., Nguyen, P., Senior, A., & Vanhoucke, V. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-10 · Cited by 243
Kingsbury, B., Sainath, T. N., & Soltau, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-3 · Cited by 220
Adaptation of context-dependent deep neural networks for automatic speech recognition
Yao, K., Yu, D., Seide, F., Su, H., Deng, L., & Gong, Y. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424251 · Cited by 194
Self-citationYu, D., Deng, L., & Dahl, G. E. (2010) · neural information processing systems · Cited by 189
Self-citationMaking Deep Belief Networks effective for large vocabulary continuous speech recognition
Sainath, T. N., Kingsbury, B., Ramabhadran, B., Fousek, P., Novak, P., & Mohamed, A.-R. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163900 · Cited by 189
Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system
Neto, J., Almeida, L., Hochberg, M., Martins, C., Nunes, L., Renals, S., & Robinson, T. (1995) · 4th European Conference on Speech Communication and Technology (Eurospeech 1995) · doi:10.21437/eurospeech.1995-411 · Cited by 175
Linear hidden transformations for adaptation of hybrid ANN/HMM models
Gemello, R., Mana, F., Scanzio, S., Laface, P., & Mori, R. D. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.005 · Cited by 162
Li, B., & Sim, K. C. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-214 · Cited by 155
Su, H., Li, G., Yu, D., & Seide, F. T. B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638951 · Cited by 139
Self-citationConversational speech transcription using context-dependent deep neural networks
Yu, D., Seide, F. T. B., & Li, G. (2012) · International Conference on Machine Learning · Cited by 131
Self-citationThe Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition
Yu, D., Deng, L., & Seide, F. T. B. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2227738 · Cited by 114
Self-citationRegularized Adaptation of Discriminative Classifiers
Li, X., & Bilmes, J. A. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2006.1660001 · Cited by 80
Connectionist speaker normalization and adaptation
Abrash, V., Franco, H., Sankar, A., & Cohen, M. (1995) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1995-414 · Cited by 70
Automated directory assistance system - from theory to practice
Yu, D., Ju, Y., Wang, Y., Zweig, G., & Acero, A. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-65 · Cited by 52
Self-citationA Novel Framework and Training Algorithm for Variable-Parameter Hidden Markov Models
Yu, D., Deng, L., Gong, Y., & Acero, A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2020890 · Cited by 41
Self-citationFACTORIZED DEEP NEURAL NETWORKS FOR ADAPTIVE SPEECH RECOGNITION
Yu, D., Chen, X., & Deng, L. (2012) · Cited by 25
Self-citationAdaptation of Artificial Neural Networks Avoiding Catastrophic Forgetting
Albesano, D., Gemello, R., Laface, P., Mana, F., & Scanzio, S. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2006.1716291 · Cited by 20
Cited by
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
Self-citationDeep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Deep learning for smart manufacturing: Methods and applications
Wang, J., Ma, Y., Zhang, L., Gao, R. X., & Wu, D. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.01.003 · Cited by 1,740
Speech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I., Attili, I., Azzeh, M., & Shaalan, K. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,152
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801
Self-citation
Metadata
- Authors
- Yu, D., Yao, K., Su, H., Li, G., & Seide, F.
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 7893-7897
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W1989549063
- Cited by
- 408
- References
- 25
- FWCI
- 31.87
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.