Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Abstract
Long Short-Term Memory (LSTM) is a specific recurrent neural network (RNN) architecture that was designed to model temporal sequences and their long-range dependencies more accurately than conventional RNNs. In this paper, we explore LSTM RNN architectures for large scale acoustic modeling in speech recognition. We recently showed that LSTM RNNs are more effective than DNNs and conventional RNNs for acoustic modeling, considering moderately-sized models trained on a single machine. Here, we introduce the first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines. We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance. This architecture makes more effective use of model parameters than the others considered, converges quickly, and outperforms a deep feed forward neural network having an order of magnitude more parameters. Index Terms: Long Short-Term Memory, LSTM, recurrent neural network, RNN, speech recognition, acoustic modeling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep neural networks have been widely used and achieved excellent performance in many applications including speech recognition, face recognition, image classification, object detection, semantic segmentation, and natural language processing.
“Deep neural networks (DNNs) have been widely used in many areas and have achieved excellent performance in many applications including speech recognition [24], face recognition [25], image classification [26], object detection [27], semantic segmentation [28], natural language processing [29]”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · see citing works
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680 · see citing works
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993 · see citing works
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
Karpatne, A., Atluri, G., Faghmous, J. H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., & Kumar, V. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2720168 · Cited by 1,752 · see citing works
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093 · see citing works
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100 · see citing works
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928 · see citing works
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Xie, J., Yu, F. R., Huang, T., Xie, R., Liu, J., Wang, C., & Liu, Y. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2866942 · Cited by 700 · see citing works
Deep learning in video multi-object tracking: A survey
Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., & Herrera, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.023 · Cited by 665 · see citing works
Zheng, H., Yuan, J., & Chen, L. (2017) · Energies · doi:10.3390/en10081168 · Cited by 635 · see citing works
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601 · see citing works
ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., Wang, Y., et al. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021745 · Cited by 591 · see citing works
Passive Detection of Atrial Fibrillation Using a Commercially Available Smartwatch
Tison, G. H., Sanchez, J. M., Ballinger, B., Singh, A., Olgin, J. E., Pletcher, M. J., Vittinghoff, E., Lee, E. S., Fan, S. M., Gladstone, R. A., et al. (2018) · JAMA Cardiology · doi:10.1001/jamacardio.2018.0136 · Cited by 566 · see citing works
Video-based emotion recognition using CNN-RNN and C3D hybrid networks
Fan, Y., Lu, X., Li, D., & Liu, Y. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997632 · Cited by 556 · see citing works
The Microsoft 2017 Conversational Speech Recognition System
Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., & Stolcke, A. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461870 · Cited by 472 · see citing works
Dissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · see citing works
A survey on deep learning-based fine-grained object classification and semantic segmentation
Zhao, B., Feng, J., Wu, X., & Yan, S. (2017) · International Journal of Automation and Computing · doi:10.1007/s11633-017-1053-3 · Cited by 339 · see citing works
The microsoft 2016 conversational speech recognition system
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953159 · Cited by 326 · see citing works
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304 · see citing works
Self-citationSAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG
Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., & Xu, X. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00037 · Cited by 269 · see citing works
Unsupervised Pretraining for Sequence to Sequence Learning
Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251 · see citing works
Toward Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., & Zweig, G. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2756440 · Cited by 245 · see citing works
Uddin, M. Z., Hassan, M. M., Alsanad, A., & Savaglio, C. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.08.004 · Cited by 194 · see citing works
Jung, D.-H., Kim, H. S., Jhin, C., Kim, H.-J., & Park, S. H. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105402 · Cited by 185 · see citing works
Current trends in the development of intelligent unmanned autonomous systems
Zhang, T., Li, Q., Zhang, C.-S., Liang, H.-W., Li, P., Wang, T.-M., Li, S., Zhu, Y.-L., & Wu, C. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601650 · Cited by 184 · see citing works
A survey on deep transfer learning to edge computing for mitigating the COVID-19 pandemic
Sufian, A., Ghosh, A., Sadiq, A. S., & Smarandache, F. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2020.101830 · Cited by 156 · see citing works
ST-DeepHAR: Deep Learning Model for Human Activity Recognition in IoHT Applications
Abdel-Basset, M., Hawash, H., Chakrabortty, R. K., Ryan, M., Elhoseny, M., & Song, H. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3033430 · Cited by 136 · see citing works
LSTM time and frequency recurrence for automatic speech recognition
Li, J., Mohamed, A., Zweig, G., & Gong, Y. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404793 · Cited by 119 · see citing works
Pastur-Romay, L., Cedrón, F., Pazos, A., & Porto-Pazos, A. (2016) · International Journal of Molecular Sciences · doi:10.3390/ijms17081313 · Cited by 109 · see citing works
Factorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · see citing works
“We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”
MethodInfluentialData augmentation for self-paced motor imagery classification with C-LSTM.
Freer, D., & Yang, G.-Z. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab57c0 · Cited by 96 · see citing works
Li, Y., Bao, T., Chen, H., Zhang, K., Shu, X., Chen, Z., & Hu, Y. (2021) · Measurement · doi:10.1016/j.measurement.2021.109377 · Cited by 90 · see citing works
Recurrent and Dynamic Models for Predicting Streaming Video Quality of Experience
Bampis, C. G., Li, Z., Katsavounidis, I., & Bovik, A. C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2815842 · Cited by 84 · see citing works
Yanamandra, K., Chen, G. L., Xu, X., Mac, G., & Gupta, N. (2020) · Composites Science and Technology · doi:10.1016/j.compscitech.2020.108318 · Cited by 72 · see citing works
A co-design framework of neural networks and quantum circuits towards quantum advantage
Jiang, W., Xiong, J., & Shi, Y. (2021) · Nature Communications · doi:10.1038/s41467-020-20729-5 · Cited by 63 · see citing works
Reflective fiber fault detection and characterization using long short-term memory
Abdelli, K., Grießer, H., Ehrle, P., Tropschug, C., & Pachnicke, S. (2021) · Journal of Optical Communications and Networking · doi:10.1364/jocn.423625 · Cited by 53 · see citing works
Personalization of end-to-end speech recognition on mobile devices for named entities
Sim, K. C., Beaufays, F., Benard, A., Guliani, D., Kabel, A., Khare, N., Lucassen, T., Zadrazil, P., Zhang, H., Johnson, L., et al. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9003775 · Cited by 54 · see citing works
Self-citationMikus, A., Hoogendoorn, M., Rocha, A., Gama, J., Ruwaard, J., & Riper, H. (2018) · Internet Interventions · doi:10.1016/j.invent.2017.10.001 · Cited by 31 · see citing works
Moirangthem, D. S., & Lee, M. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.12.022 · Cited by 30 · see citing works
Automatic Speech Assessment for People with Aphasia Using TDNN-BLSTM with Multi-Task Learning
Qin, Y., Lee, T., Feng, S., & Kong, A. P. H. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1630 · Cited by 30 · see citing works
Quantum deep learning neural networks
Kamruzzaman, A., Alhwaiti, Y., Leider, A., & Tappert, C. C. (2020) · Lecture notes in networks and systems · doi:10.1007/978-3-030-12385-7_24 · Cited by 26 · see citing works
Reducing the Computational Complexity of Two-Dimensional LSTMs
Li, B., & Sainath, T. N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1164 · Cited by 13 · see citing works
Evolving Long Short-Term Memory Networks
Lobo Neto, V. C., Passos, L. A., & Papa, J. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-50417-5_25 · Cited by 9 · see citing works
Design and evaluation of a global workspace agent embodied in a realistic multimodal environment
Dossa, R. F. J., Arulkumaran, K., Juliani, A., Sasai, S., & Kanai, R. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1352685 · Cited by 5 · see citing works
“…studies between populations of neurons and their artificial analogs (Bedia et al., 2007; Li and Fan, 2019), such as recurrent neural networks (RNNs), which are a prominent class of architecture used to process temporal data (Hochreiter and Schmidhuber, 1997; Cho et al., 2014; Sak et al., 2014).”
Background
- Authors
- Sak, H., Senior, A., & Beaufays, F.
- ORCID
- Senior, Andrew
- Venue
- Interspeech 2014
- Publisher
- ISCA
- Published
- 2014-09-14
- Pages
- 338-342
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2293634267
- Cited by
- 2,973
- References
- 2
- Citations by year
- 2026: 78 · 2025: 159 · 2024: 223 · 2023: 285 · 2022: 315 · 2021: 361 · 2020: 406 · 2019: 365 · 2018: 339 · 2017: 231 · 2016: 143 · 2015: 61 · 2014: 7
- FWCI
- 86.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- long short-term memory, recurrent neural networks, acoustic modeling, speech recognition, distributed training, asynchronous stochastic gradient descent
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
