CoChat
Sign Up
PreprintCited by 7,073Neural Networks and ApplicationsSpeech Recognition and SynthesisTopic Modeling

LSTM: A Search Space Odyssey

Open access
Greff, Klaus, Srivastava, Rupesh Kumar, Koutník, Jan, Steunebrink, Bas R., Schmidhuber, Jürgen·2015·arXiv
DOI 10.48550/arxiv.1503.04069arXiv 1503.04069
Open this paper in CoChat — free accountOpen PDF

Abstract

Several variants of the Long Short-Term Memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework. In total, we summarize the results of 5400 experimental runs ($\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1503.04069

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ablation studies showed that all components of the LSTM memory cell are crucial.

    “Ablation studies showed that all components of the memory cell are crucial (Greff et al., 2015).”

    From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Greff et al., 2015

    The Comparison of the Variants section reports that coupling the input and forget gates and removing peephole connections did not significantly change mean performance. The Conclusion likewise says these changes simplified LSTMs without significantly decreasing performance. The paper identifies the forget gate and output activation function as the most critical components, not all components as crucial. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Sepp Hochreiter. Untersuchungen zu dynamischen neu- ronalen Netzen. Masters Thesis, Technische Universität München, München, 1991.
  2. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
  3. A Graves, M Liwicki, S Fernandez, R Bertolami, H Bunke, and J Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 2009.
  4. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves Recurrent Neural Networks for Handwriting Recognition. arXiv:1312.4569 [cs], November 2013. URL http://arxiv.org/abs/1312. 4569. · arXiv
  5. Patrick Doetsch, Michal Kozielski, and Hermann Ney. Fast and robust training of recurrent neural networks for offline handwriting recognition. In 14th International Conference on Frontiers in Handwriting Recognition, 2014. URL http://people.sabanciuniv.edu/berrin/cs581/ Papers/icfhr2014/data/4334a279.pdf.
  6. Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs], August 2013. URL http://arxiv.org/abs/1308.0850. · arXiv
  7. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Re- current Neural Network Regularization. arXiv:1409.2329 [cs], September 2014. URL http://arxiv.org/abs/1409. 2329. · arXiv
  8. Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the Rare Word Problem in Neural Machine Translation. arXiv preprint arXiv:1410.8206, 2014. URL http://arxiv.org/abs/1410. 8206. · arXiv
  9. Hasim Sak, Andrew Senior, and Franc ¸oise Beaufays. Long short-term memory recurrent neural network ar- chitectures for large scale acoustic modeling. In Pro- ceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), 2014. URL http://193.6.4.39/ ∼ czap/letoltes/IS14/IS2014/ PDF/AUTHOR/IS141304.PDF.
  10. Yuchen Fan, Yao Qian, Fenglong Xie, and Frank K. Soong. TTS synthesis with bidirectional LSTM based recurrent neural networks. In Proc. Interspeech, 2014.
  11. Søren Kaae Sønderby and Ole Winther. Protein Secondary Structure Prediction with Long Short Term Memory Networks. arXiv:1412.7828 [cs, q-bio], December 2014. URL http://arxiv.org/abs/1412.7828. arXiv: 1412.7828. · arXiv
  12. E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, and B. Schuller. Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2164-2168, May 2014. doi: 10.1109/ICASSP.2014. 6853982. · DOI
  13. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term Recurrent Convolu- tional Networks for Visual Recognition and Descrip- tion. arXiv:1411.4389 [cs], November 2014. URL http://arxiv.org/abs/1411.4389. arXiv: 1411.4389. · arXiv
  14. Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Term Memory. Neural Computation, 9(8):1735-1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco. 1997.9.8.1735. URL http://www.bioinf.jku.at/publications/ older/2604.pdf. · DOI · Paper page
  15. R. L. Anderson. Recent Advances in Finding Best Operating Conditions. Journal of the American Statistical Association, 48(264):789-798, December 1953. ISSN 0162-1459. doi: 10.2307/2281072. URL http://www.jstor. org/stable/2281072. · DOI
  16. Francisco J. Solis and Roger J.-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1):19-30, February 1981. ISSN 0364-765X. doi: 10.1287/moor.6.1.19. URL http://pubsonline.informs. org/doi/abs/10.1287/moor.6.1.19. · DOI
  17. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281-305, 2012. URL http: //dl.acm.org/citation.cfm?id=2188395.
  18. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An Efficient Approach for Assessing Hyperparameter Importance. pages 754-762, 2014. URL http://jmlr.org/ proceedings/papers/v32/hutter14.html.
  19. Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6): 602-610, July 2005. ISSN 0893-6080. doi: 10.1016/j. neunet.2005.06.042. URL http://www.sciencedirect.com/ science/article/pii/S0893608005001206. · DOI
  20. Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 2, pages 850-855, 1999.
  21. Felix A. Gers and Jürgen Schmidhuber. Recurrent nets that time and count. In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, volume 3, pages 189-194. IEEE, 2000. ISBN 0769506194.
  22. AJ Robinson and Frank Fallside. The utility driven dynamic error propagation network. University of Cambridge Department of Engineering, 1987.
  23. Felix A. Gers, Juan Antonio Pérez-Ortiz, Douglas Eck, and Jürgen Schmidhuber. DEFK-LSTM. In ESANN 2002, Proceedings of the 10th Eurorean Symposium on Artificial Neural Networks, 2002.
  24. J Schmidhuber, D Wierstra, M Gagliolo, and F J Gomez. Training Recurrent Networks by EVOLINO. Neural Computation, 19(3):757-779, 2007.
  25. Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. Evolving memory cell structures for sequence learning. In Artificial Neural Networks-ICANN 2009, pages 755-764. Springer, 2009. URL http://link. springer.com/chapter/10.1007/978-3-642-04277-5 76. · DOI
  26. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network archi- tectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342- 2350, 2015.
  27. Sebastian Otte, Marcus Liwicki, and Andreas Zell. Dy- namic Cortex Memory: Enhancing Recurrent Neural Net- works for Gradient-Based Sequence Learning. In Artificial Neural Networks and Machine Learning -ICANN 2014, number 8681 in Lecture Notes in Computer Science, pages 1-8. Springer International Publishing, September 2014. ISBN 978-3-319-11178-0, 978-3-319-11179-7. URL http://link.springer.com/chapter/10.1007/978-3-319- 11179-7 1. · DOI
  28. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078, 2014. URL http://arxiv.org/ abs/1406.1078. · arXiv
  29. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs], December 2014. URL http://arxiv. org/abs/1412.3555. · arXiv
  30. David Crystal. Dictionary of linguistics and phonetics, volume 30. John Wiley & Sons, 2011.
  31. P. Mermelstein. Distance measures for speech recognition: Psychological and instrumental. In C. H. Chen, editor, Pattern Recognition and Artificial Intelligence, pages 374- 388. Academic Press, New York, 1976.
  32. Andrew K. Halberstadt. Heterogeneous acoustic mea- surements and multiple classifiers for speech recognition. PhD thesis, Massachusetts Institute of Technology, 1998.
  33. Marcus Liwicki and Horst Bunke. IAM-OnDB-an on-line English sentence database acquired from handwritten text on a whiteboard. In Document Analysis and Recognition, 2005. Proceedings. Eighth International Conference on, pages 956-961. IEEE, 2005.
  34. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classifica- tion: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006. URL http://dl.acm.org/citation.cfm?id=1143891.
  35. Moray Allan and Christopher KI Williams. Harmonising chorales by probabilistic inference. Advances in neural information processing systems, 17:25-32, 2005.
  36. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription. pages 1159-1166, 2012. URL http://icml.cc/discuss/2012/590.html.

Cited by

Sorted by citations

Authors
Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J.
Affiliations
University of Applied Sciences and Arts of Southern Switzerland (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
Volume
28
Issue
10
Pages
2222-2232
ISSN
2162-237X (ISSN-L)
DOI
10.48550/arxiv.1503.04069
arXiv
1503.04069
OpenAlex
W1689711448
Cited by
7,073
References
0
Citations by year
2026: 650 · 2025: 836 · 2024: 821 · 2023: 821 · 2022: 882 · 2021: 874 · 2020: 774 · 2019: 695 · 2018: 401 · 2017: 193 · 2016: 101 · 2015: 24
FWCI
347.08
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (138219, 156682)
  • European Institute of Innovation and Technology (FP7-ICT-288551, FP7-ICT-270247, FP7-ICT-317662)
  • FP7 Information and Communication Technologies
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
long short-term memory, recurrent neural networks, hyperparameter optimization, random search, functional analysis of variance, forget gate, speech recognition, handwriting recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review