The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
LSTM: A Search Space Odyssey
Open accessAbstract
Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( ≈ 15 years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“NLP FL is applied in next-word prediction in mobile keyboards by adopting the FedAvg algorithm to learn CIFG [93].[28,[94][95][96][97]”
From Applications of Federated Learning; Taxonomy, Challenges, and Research Trends (2022) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Self-citationLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152
Self-citationGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Self-citationLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
(2013) · Cited by 3,740
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 3,174
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
A Dictionary of Linguistics and Phonetics
Major, R. C., & Crystal, D. S. (1992) · Modern Language Journal · doi:10.2307/330198 · Cited by 2,766
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · International Conference on Artificial Neural Networks · doi:10.1049/cp:19991218 · Cited by 2,467
A Dictionary of Linguistics and Phonetics
Crystal, D. S. (2008) · Wiley eBooks · doi:10.1002/9781444302776 · Cited by 2,199
A Novel Connectionist System for Unconstrained Handwriting Recognition
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Horst Bunke, A., & Schmidhuber, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.137 · Cited by 2,041
Self-citationMinimization by Random Search Techniques
Solís, F. J., & Wets, R. J. ‐. (1981) · Mathematics of Operations Research · doi:10.1287/moor.6.1.19 · Cited by 1,691
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L., Fisher, W. M., Fiscus, J. G., Pallett, D., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,533
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · Defense Technical Information Center (DTIC) · doi:10.21236/ada623249 · Cited by 1,101
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 947
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter, S., Bengio, Y., Informatik, F. F., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001) · Cited by 863
Self-citationRecurrent nets that time and count
Gers, F. A., & Schmidhuber, J. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.861302 · Cited by 689
Self-citationAddressing the Rare Word Problem in Neural Machine Translation
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2015) · doi:10.3115/v1/p15-1002 · Cited by 626
Dropout improves Recurrent Neural Networks for Handwriting RecognitionPreprint
(2016) · Cited by 533
Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6392 · Cited by 484
TTS synthesis with bidirectional LSTM based recurrent neural networks
Fan, Y., Qian, Y., Xie, F., & Soong, F. K. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-443 · Cited by 445
An Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · Cited by 350
Distance measures for speech recognition, psychological and instrumental
Mermelstein, P. G. (1976) · Medical Entomology and Zoology · Cited by 315
Training Recurrent Networks by Evolino
Schmidhuber, J., Wierstra, D., Gagliolo, M., & Gomez, F. J. (2007) · Neural Computation · doi:10.1162/neco.2007.19.3.757 · Cited by 253
Self-citationUnconstrained On-line Handwriting Recognition with Recurrent Neural Networks
Graves, A., Liwicki, M., Horst Bunke, A., Schmidhuber, J., & Fernández, S. (2007) · Bern Open Repository and Information System (University of Bern) · Cited by 222
Self-citationGeneralized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables
Hooker, G. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x237892 · Cited by 218
IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard
Liwicki, M., & Bunke, H. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2005.132 · Cited by 212
Fast and Robust Training of Recurrent Neural Networks for Offline Handwriting Recognition
Doetsch, P., Kozielski, M., & Ney, H. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.54 · Cited by 169
Harmonising Chorales by Probabilistic Inference
Allan, M., & Williams, C. K. I. (2004) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 136
Protein Secondary Structure Prediction with Long Short Term Memory NetworksPreprint
Sønderby, S. K., & Winther, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.7828 · Cited by 129
Evolving Memory Cell Structures for Sequence Learning
Bayer, J., Wierstra, D., Togelius, J., & Schmidhuber, J. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04277-5_76 · Cited by 105
Self-citationAddressing the Rare Word Problem in Neural Machine TranslationPreprint
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.8206 · Cited by 101
Dropout Improves Recurrent Neural Networks for Handwriting Recognition
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.55 · Cited by 99
Marchi, E., Ferroni, G., Eyben, F., Gabrielli, L., Squartini, S., & Schuller, B. W. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853982 · Cited by 94
Heterogeneous acoustic measurements and multiple classifiers for speech recognition
Halberstadt, A. K., & Glass, J. (1999) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 89
Recent Advances in Finding Best Operating Conditions
Anderson, R. L. (1953) · Journal of the American Statistical Association · doi:10.1080/01621459.1953.10501200 · Cited by 64
Artificial Neural Networks and Machine Learning – ICANN 2014
Wermter, S., Villa, A. E. P., Palm, G., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7 · Cited by 40
Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning
Otte, S., Liwicki, M., & Zell, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7_1 · Cited by 25
Recent Advances in Finding Best Operating Conditions
Anderson, R. L. (1953) · Journal of the American Statistical Association · doi:10.2307/2281072 · Cited by 18
Long-term Recurrent Convolutional Networks for Visual Recognition and DescriptionPreprint
Donahue, J., Hendricks, L. A., Rohrbach, M., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4389 · Cited by 12
Artificial Neural Networks – ICANN 2009
Alippi, C., Polycarpou, M., Ellinas, G., et al. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04277-5 · Cited by 8
Barbero (2014) · doi:10.1016/j · Cited by 1
Dropout improves Recurrent Neural Networks for Handwriting Recognition
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2013) · arXiv · doi:10.48550/arxiv.1312.4569
Flagged first, then by citations
Research works printed but not linked: 2
- gers, 2002, DEKF-LSTM, Proc European Symp Artificial Neural Networks (ESANN), 369
- robinson, 1987, The utility driven dynamic error propagation network
Cited by
Sorted by citations
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,870
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
A review on the long short-term memory model
Houdt, G. V., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 2,198
Koroniotis, N., Moustafa, N., Sitnikova, E., & Turnbull, B. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.05.041 · Cited by 1,983
Metadata
- Authors
- Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J.
- Affiliations
- University of Applied Sciences and Arts of Southern Switzerland (CH)
- ORCID
- Greff, Klaus
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2017-10
- Volume
- 28
- Issue
- 10
- Pages
- 2222-2232
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W1689711448
- Cited by
- 6,955
- References
- 53
- Citations by year
- 2026: 602 · 2025: 812 · 2024: 810 · 2023: 822 · 2022: 872 · 2021: 881 · 2020: 778 · 2019: 686 · 2018: 383 · 2017: 188
- FWCI
- 370.42
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Swiss National Science Foundation through the Project Theory and Practice of Reinforcement Learning 2, and through the Project Advanced Reinforcement Learning (138219, 156682)
- European Institute of Innovation and Technology through the Project Neural Dynamics, through the Project NASCENCE and through the Project WAY (FP7-ICT-270247, FP7-ICT-317662, FP7-ICT-288551)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- long short-term memory, recurrent neural networks, hyperparameter optimization, random search, functional analysis of variance, forget gate, speech recognition, handwriting recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
