Improving deep neural networks for LVCSR using rectified linear units and dropout
Abstract
Recently, pre-trained deep neural networks (DNNs) have outperformed traditional acoustic models based on Gaussian mixture models (GMMs) on a variety of large vocabulary speech recognition benchmarks. Deep neural nets have also achieved excellent results on various computer vision tasks using a random “dropout” procedure that drastically improves generalization error by randomly omitting a fraction of the hidden units in all layers. Since dropout helps avoid over-fitting, it has also been successful on a small-scale phone recognition task using larger neural nets. However, training deep neural net acoustic models for large vocabulary speech recognition takes a very long time and dropout is likely to only increase training time. Neural networks with rectified linear unit (ReLU) non-linearities have been highly successful for computer vision tasks and proved faster to train than standard sigmoid units, sometimes also improving discriminative performance. In this work, we show on a 50-hour English Broadcast News task that modified deep neural networks using ReLUs trained with dropout during frame level training provide an 4.2% relative improvement over a DNN trained with sigmoid units, and a 14.4% relative improvement over a strong GMM/HMM system. We were able to obtain our results with minimal human hyper-parameter tuning using publicly available Bayesian optimization code.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Despite its widespread use, the KDD98 dataset has limitations in terms of its reflection on real-world scenarios.
“Despite its widespread use, it has limitations in terms of its reflection on real-world scenarios.(Creech & Hu, 2014b) [70].”
From Comparative Analysis of Intrusion Detection Systems and Machine Learning Based Model Analysis through Decision Tree (2023) · cites this paper as [70]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Self-citation(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Self-citationContext-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · see citing works
Self-citationWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · Cited by 1,839 · see citing works
Self-citationAcoustic Modeling Using Deep Belief Networks
Mohamed, A., Dahl, G. E., & Hinton, G. E. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,731 · see citing works
Self-citationLearning a better representation of speech soundwaves using restricted boltzmann machines
Jaitly, N., & Hinton, G. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947700 · Cited by 231 · cited 2× · see citing works
“the only use of these units in speech to date was in [10] where Gaussian-ReLU RBMs were used to learn features from raw, unlabeled acoustic data to later use in phone recognition experiments on TIMIT.”
MethodSelf-citationKingsbury, B. E. D., Sainath, T. N., & Soltau, H. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-3 · Cited by 222 · see citing works
Self-citationMaking Deep Belief Networks effective for large vocabulary continuous speech recognition
Sainath, T. N., Kingsbury, B. E. D., Ramabhadran, B., Fousek, P., Novak, P., & Mohamed, A. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163900 · Cited by 192 · see citing works
Self-citationAuto-encoder bottleneck features using deep belief networks
Sainath, T. N., Kingsbury, B. E. D., & Ramabhadran, B. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288833 · Cited by 185 · see citing works
Self-citationThe IBM Attila speech recognition toolkit
Soltau, H., Saon, G., & Kingsbury, B. E. D. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700829 · Cited by 119 · see citing works
CUDAMat: a CUDA-based matrix class for Python
Mnih, V. (2009) · Cited by 68 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672 · see citing works
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414 · see citing works
End-to-end encrypted traffic classification with one-dimensional convolution neural networks
Wang, W., Zhu, M., Wang, J., Zeng, X., & Yang, Z. (2017) · 2017 IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi.2017.8004872 · Cited by 991 · see citing works
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834 · see citing works
Medical image classification with convolutional neural network
Li, Q., Cai, W., Wang, X., Zhou, Y., Feng, D. D., & Chen, M. (2014) · 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV) · doi:10.1109/icarcv.2014.7064414 · Cited by 880 · see citing works
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821 · see citing works
Robust Face Recognition via Multimodal Deep Face Representation
Ding, C., & Tao, D. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477042 · Cited by 449 · see citing works
Deep residual learning-based fault diagnosis method for rotating machinery
Zhang, W., Li, X., & Ding, Q. (2019) · ISA Transactions · doi:10.1016/j.isatra.2018.12.025 · Cited by 416 · see citing works
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · see citing works
“features from co-adapting, and it has been reported to yield good results when training deep neural networks (Krizhevsky et al., 2012; Dahl et al., 2013).”
BackgroundMethodAutomatic fruit classification using deep learning for industrial applications
Hossain, M. S., Al-Hammadi, M., & Muhammad, G. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2875149 · Cited by 347 · see citing works
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334 · see citing works
Neural network with unbounded activation functions is universal approximator
Sonoda, S., & Murata, N. (2017) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2015.12.005 · Cited by 303 · see citing works
Azam, Z., Islam, M. M., & Huda, M. N. (2023) · IEEE Access · doi:10.1109/access.2023.3296444 · Cited by 265 · see citing works
Deep learning based on Batch Normalization for P300 signal detection
Liu, M., Wu, W., Gu, Z., Yu, Z., Qi, F., & Li, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.08.039 · Cited by 230 · see citing works
SPOT-Disorder2: Improved Protein Intrinsic Disorder Prediction by Ensembled Deep Learning
Hanson, J., Paliwal, K. K., Litfin, T., & Zhou, Y. (2019) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2019.01.004 · Cited by 165 · see citing works
Zhou, H.-Y., Chen, X., Zhang, Y., Luo, R., Wang, L., & Yu, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-021-00425-9 · Cited by 126 · see citing works
The many-body expansion combined with neural networks
Yao, K., Herr, J. E., & Parkhill, J. (2017) · The Journal of Chemical Physics · doi:10.1063/1.4973380 · Cited by 120 · see citing works
Benou, A., Veksler, R., Friedman, A., & Riklin Raviv, T. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.006 · Cited by 91 · see citing works
Tsuji, S., Hase, T., Yachie-Kinoshita, A., Nishino, T., Ghosh, S., Kikuchi, M., Shimokawa, K., Aburatani, H., Kitano, H., & Tanaka, H. (2021) · Alzheimer's Research & Therapy · doi:10.1186/s13195-021-00826-3 · Cited by 75 · see citing works
Robust application of new deep learning tools: an experimental study in medical imaging
Alzubaidi, L., Fadhel, M. A., Al-Shamma, O., Zhang, J., Santamaría, J., & Duan, Y. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10942-9 · Cited by 68 · see citing works
Towards perturbation prediction of biological networks using deep learning
Li, D., & Gao, J. (2019) · Scientific Reports · doi:10.1038/s41598-019-48391-y · Cited by 16 · see citing works
An Instruction Set Architecture for Machine Learning
Chen, Y., Lan, H., Du, Z., Liu, S., Tao, J., Han, D., Luo, T., Guo, Q., Li, L., Xie, Y., et al. (2018) · ACM Transactions on Computer Systems · doi:10.1145/3331469 · Cited by 14 · see citing works
Kim, J., Kim, H., Kim, K., & Ahn, J. M. (2023) · Water · doi:10.3390/w15122293 · Cited by 11 · see citing works
- Authors
- Dahl, G. E., Sainath, T. N., & Hinton, G. E.
- Affiliations
- University of Toronto (CA)
- IBM (United States) (US)
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 8609-8613
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2062227835
- Cited by
- 1,310
- References
- 16
- Citations by year
- 2026: 14 · 2025: 33 · 2024: 45 · 2023: 82 · 2022: 118 · 2021: 179 · 2020: 199 · 2019: 179 · 2018: 137 · 2017: 87 · 2016: 92 · 2015: 79 · 2014: 58 · 2013: 8
- FWCI
- 51.02
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- large vocabulary speech recognition, deep neural network, rectified linear units, dropout, acoustic modeling, broadcast news, overfitting, Bayesian optimization, hyperparameter optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
