Improvements to Deep Convolutional Neural Networks for LVCSR
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a variety of LVCSR tasks. In this paper, we describe different methods to further improve CNN performance. First, we conduct a deep analysis comparing limited weight sharing and full weight sharing with state-of-the-art features. Second, we apply various pooling strategies that have shown improvements in computer vision to an LVCSR speech task. Third, we introduce a method to effectively incorporate speaker adaptation, namely fMLLR, into log-mel features. Fourth, we introduce an effective strategy to use dropout during Hessian-free sequence training. We find that with these improvements, particularly with fMLLR and dropout, we are able to achieve an additional 2-3% relative improvement in WER on a 50-hour Broadcast News task over our previous best CNN baseline. On a larger 400-hour BN task, we find an additional 4-5% relative improvement over our previous best CNN baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationImproving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · cited 2×
“Dropout is a popular technique to prevent over-fitting during neural network training [16].”
BackgroundMethodPhoneme recognition using time-delay neural networks
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.21701 · Cited by 2,643 · cited 1×
“Most previous CNN work in speech explored pooling in frequency only ([4], [3], [5]), though [13] did investigate CNNs with pooling in time, but not frequency.”
BackgroundMaximum likelihood linear transformations for HMM-based speech recognition
Gales, M. J. F. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0043 · Cited by 1,482 · cited 2×
“However, speaker adapted features, such as feature space maximum likelihood linear regression (fMLLR) features [10], typically give the best performance for DNNs.”
BackgroundMethodLearning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · cited 4×
“In [12], ReLU+dropout was shown to give good performance for cross-entropy (CE) trained DNNs but was not employed during HF sequence-training.”
BackgroundMethodResultInfluentialSelf-citationDeep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A.-R., Kingsbury, B., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,053 · cited 10×
“The CNN architecture proposed in [4] was a somewhat vanilla architecture that had been used in computer vision for many years.”
BackgroundMethodResultInfluentialSelf-citationApplying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288864 · Cited by 876 · cited 6×
“Thus, most work with LWS up to this point has looked at LWS with one layer [3], [5].”
BackgroundMethodInfluentialSelf-citationStochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755 · cited 3×
“Stochastic pooling has also shown huge improvements in error rate in computer vision [7].”
BackgroundMethodDeep learning via Hessian-free optimization
Martens, J. (2010) · International Conference on Machine Learning · Cited by 732
Cited by
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
“[306] combined dropout with a number of novel techniques described in this section (including the use of deep CNNs, Hessian-free sequence learning, the use of ReLU units, and the use of joint fMLLR and filterbank features, etc.”
BackgroundMethodDeep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
Deep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678
Self-citationSpeech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I., Attili, I., Azzeh, M., & Shaalan, K. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,152
A review of convolutional neural networks in computer vision
Zhao, X., Wang, L., Zhang, Y., Han, X., Deveci, M., & Parmar, M. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10721-6 · Cited by 1,148
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T., Schluter, J., Chang, S.-Y., & Sainath, T. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 855
Self-citationConceptual Understanding of Convolutional Neural Network- A Deep Learning Approach
Indolia, S., Goswami, A. K., Mishra, S. P., & Asopa, P. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.05.069 · Cited by 748
Jing, L., Zhao, M., Li, P., & Xu, X. (2017) · Measurement · doi:10.1016/j.measurement.2017.07.017 · Cited by 698
Deep Convolutional Neural Networks for Large-scale Speech Tasks
Sainath, T. N., Kingsbury, B., Saon, G., Soltau, H., Mohamed, A.-R., Dahl, G., & Ramabhadran, B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.08.005 · Cited by 496
Self-citationLearning the speech front-end with raw waveform CLDNNs
Sainath, T. N., Weiss, R. J., Senior, A., Wilson, K. W., & Vinyals, O. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-1 · Cited by 430
Self-citationVery deep convolutional networks for end-to-end speech recognition
Zhang, Y., Chan, W., & Jaitly, N. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953077 · Cited by 407
Recommendation system based on deep learning methods: a systematic review and new directions
Da’u, A., & Salim, N. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09744-1 · Cited by 402
Data Augmentation for Deep Neural Network Acoustic Modeling
Cui, X., Goel, V., & Kingsbury, B. (2015) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2015.2438544 · Cited by 376
“2 shows a mean and speaker normalized LogMel feature extraction pipeline used in IBM CNN systems [31][32].”
MethodSelf-citationTowards End-to-End Speech Recognition with Deep Convolutional Neural Networks
Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Laurent, C., Bengio, Y., & Courville, A. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-1446 · Cited by 352
Fundamentals of speech recognition
Li, J., Deng, L., Haeb-Umbach, R., & Gong, Y. (2016) · Robust Automatic Speech Recognition · doi:10.1016/b978-0-12-802398-3.00002-7 · Cited by 266
Convolutional Neural Networks for Distant Speech Recognition
Swietojanski, P., Ghoshal, A., & Renals, S. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2325781 · Cited by 253
Ensemble deep learning for speech recognition
Deng, L., & Platt, J. C. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-433 · Cited by 248
Smirnov, E. A., Timoshenko, D. M., & Andrianov, S. N. (2014) · AASRI Procedia · doi:10.1016/j.aasri.2014.05.013 · Cited by 237
Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
Swietojanski, P., Li, J., & Renals, S. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2560534 · Cited by 133
“stral features, we also trained a separate set of models using 40 mel-filter-bank (FBANK) features for which fMLLR transforms cannot be easily obtained (though not impossible [64]), and for”
MethodAutomated annotation of birdsong with a neural network that segments spectrograms
Cohen, Y., Nicholson, D. A., Sanchioni, A., Mallaber, E. K., Skidanova, V., & Gardner, T. J. (2022) · eLife · doi:10.7554/elife.63853 · Cited by 80
Dysarthric Speech Recognition Using Convolutional LSTM Neural Network
Kim, M., Cao, B., An, K., & Wang, J. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-2250 · Cited by 55
New Era for Robust Speech Recognition
(2017) · doi:10.1007/978-3-319-64680-0 · Cited by 48
Metadata
- Authors
- Sainath, T. N., Kingsbury, B., Mohamed, A.-R., Dahl, G. E., Saon, G., Soltau, H., Beran, T., Aravkin, A. Y., & Ramabhadran, B.
- Venue
- 2013 IEEE Workshop on Automatic Speech Recognition and Understanding
- Publisher
- IEEE
- Published
- 2013-12
- Pages
- 315-320
- OpenAlex
- W2020683423
- Cited by
- 243
- References
- 15
- FWCI
- 17.35
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.