LSTM: A Search Space Odyssey
Open accessAbstract
Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( ≈ 15 years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FL is applied in next-word prediction in mobile keyboards by adopting the FedAvg algorithm to learn CIFG.
“NLP FL is applied in next-word prediction in mobile keyboards by adopting the FedAvg algorithm to learn CIFG [93].[28,[94][95][96][97]”
From Applications of Federated Learning; Taxonomy, Challenges, and Research Trends (2022) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Self-citationLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
(2012) · Cited by 8,009 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Self-citationGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Self-citationLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
(2013) · Cited by 3,740 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
A Dictionary of Linguistics and Phonetics
Major, R. C., & Crystal, D. S. (1992) · Modern Language Journal · doi:10.2307/330198 · Cited by 2,766 · see citing works
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509 · see citing works
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469 · see citing works
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · doi:10.1049/cp:19991218 · Cited by 2,436 · see citing works
A Dictionary of Linguistics and Phonetics
Crystal, D. S. (2008) · Wiley eBooks · doi:10.1002/9781444302776 · Cited by 2,199 · see citing works
A Novel Connectionist System for Unconstrained Handwriting Recognition
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Horst Bunke, A., & Schmidhuber, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.137 · Cited by 1,984 · see citing works
Self-citationMinimization by Random Search Techniques
Solís, F. J., & Wets, R. J. ‐. (1981) · Mathematics of Operations Research · doi:10.1287/moor.6.1.19 · Cited by 1,677 · see citing works
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547 · see citing works
DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L., Fisher, W. M., Fiscus, J. G., Pallett, D., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,533 · see citing works
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459 · see citing works
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312 · see citing works
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 947 · see citing works
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
Hochreiter, S., Bengio, Y., Informatik, F. F., Bengio, Y., Frasconi, P., & Schmidhuber, J. (2001) · Cited by 863 · see citing works
Self-citationRecurrent nets that time and count
Gers, F. A., & Schmidhuber, J. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.861302 · Cited by 656 · see citing works
Self-citationAddressing the Rare Word Problem in Neural Machine Translation
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2015) · doi:10.3115/v1/p15-1002 · Cited by 626 · see citing works
Dropout improves Recurrent Neural Networks for Handwriting RecognitionPreprint
(2016) · Cited by 514 · see citing works
Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6392 · Cited by 484 · see citing works
TTS synthesis with bidirectional LSTM based recurrent neural networks
Fan, Y., Qian, Y., Xie, F., & Soong, F. K. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-443 · Cited by 438 · see citing works
An Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · Cited by 348 · see citing works
Distance measures for speech recognition, psychological and instrumental
Mermelstein, P. G. (1976) · Medical Entomology and Zoology · Cited by 315 · see citing works
Training Recurrent Networks by Evolino
Schmidhuber, J., Wierstra, D., Gagliolo, M., & Gomez, F. J. (2007) · Neural Computation · doi:10.1162/neco.2007.19.3.757 · Cited by 248 · see citing works
Self-citationUnconstrained On-line Handwriting Recognition with Recurrent Neural Networks
Graves, A., Liwicki, M., Horst Bunke, A., Schmidhuber, J., & Fernández, S. (2007) · Bern Open Repository and Information System (University of Bern) · Cited by 222 · see citing works
Self-citationGeneralized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables
Hooker, G. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x237892 · Cited by 218 · see citing works
IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard
Liwicki, M., & Bunke, H. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2005.132 · Cited by 212 · see citing works
Fast and Robust Training of Recurrent Neural Networks for Offline Handwriting Recognition
Doetsch, P., Kozielski, M., & Ney, H. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.54 · Cited by 155 · see citing works
Harmonising Chorales by Probabilistic Inference
Allan, M., & Williams, C. K. I. (2004) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 136 · see citing works
Evolving Memory Cell Structures for Sequence Learning
Bayer, J., Wierstra, D., Togelius, J., & Schmidhuber, J. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04277-5_76 · Cited by 103 · see citing works
Self-citationAddressing the Rare Word Problem in Neural Machine TranslationPreprint
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.8206 · Cited by 101 · see citing works
Protein Secondary Structure Prediction with Long Short Term Memory NetworksPreprint
Sønderby, S. K., & Winther, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.7828 · Cited by 100 · see citing works
Marchi, E., Ferroni, G., Eyben, F., Gabrielli, L., Squartini, S., & Schuller, B. W. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853982 · Cited by 93 · see citing works
Heterogeneous acoustic measurements and multiple classifiers for speech recognition
Halberstadt, A. K., & Glass, J. (1999) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 89 · see citing works
Dropout Improves Recurrent Neural Networks for Handwriting Recognition
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.55 · Cited by 72 · see citing works
Recent Advances in Finding Best Operating Conditions
Anderson, R. L. (1953) · Journal of the American Statistical Association · doi:10.1080/01621459.1953.10501200 · Cited by 64 · see citing works
Artificial Neural Networks and Machine Learning – ICANN 2014
Wermter, S., Villa, A. E. P., Palm, G., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7 · Cited by 40 · see citing works
Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning
Otte, S., Liwicki, M., & Zell, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7_1 · Cited by 26 · see citing works
Recent Advances in Finding Best Operating Conditions
Anderson, R. L. (1953) · Journal of the American Statistical Association · doi:10.2307/2281072 · Cited by 18 · see citing works
Long-term Recurrent Convolutional Networks for Visual Recognition and DescriptionPreprint
Donahue, J., Hendricks, L. A., Rohrbach, M., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4389 · Cited by 12 · see citing works
Artificial Neural Networks – ICANN 2009
Alippi, C., Polycarpou, M., Ellinas, G., et al. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04277-5 · Cited by 8 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Dropout improves Recurrent Neural Networks for Handwriting Recognition
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2013) · arXiv · doi:10.48550/arxiv.1312.4569
Flagged first, then by citations
Research works printed but not linked: 3
- [1][1] Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen . Masters Thesis, Technische Universit¨ at M¨ unchen, M¨ unchen, 1991. -
- [23][23] AJ Robinson and Frank Fallside. The utility driven dynamic error propagation network . University of Cambridge Department of Engineering, 1987. -
- [27][27] Felix A. Gers, Juan Antonio P´ erez-Ortiz, Douglas Eck, and J¨ urgen Schmidhuber. DEFK-LSTM. In ESANN 2002, Proceedings of the 10th Eurorean Symposium on Artificial Neural Networks , 2002. -
Cited by
Sorted by citations
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680 · see citing works
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,662 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · see citing works
A review on the long short-term memory model
Houdt, G. V., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 2,066 · see citing works
Koroniotis, N., Moustafa, N., Sitnikova, E., & Turnbull, B. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.05.041 · Cited by 1,983 · see citing works
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836 · see citing works
Natural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823 · see citing works
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763 · see citing works
A State-of-the-Art Survey on Deep Learning Theory and Architectures
Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S. K. E., Sidike, P., Nasrin, M. S., Hasan, M., Essen, B. C. V., Awwal, A. A. S., & Asari, V. K. (2019) · Electronics · doi:10.3390/electronics8030292 · Cited by 1,643 · see citing works
Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks
Segler, M. H. S., Kogej, T., Tyrchan, C., & Waller, M. P. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00512 · Cited by 1,537 · see citing works
Predicting residential energy consumption using CNN-LSTM neural networks
Kim, T. Y., & Cho, S.-B. (2019) · Energy · doi:10.1016/j.energy.2019.05.230 · Cited by 1,492 · see citing works
Villarraga Ossa, E. F. (2021) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1,472 · see citing works
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459 · see citing works
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · see citing works
Federated Learning for Mobile Keyboard Prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., & Ramage, D. (2018) · arXiv · doi:10.48550/arxiv.1811.03604 · Cited by 1,148 · see citing works
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,182 · see citing works
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
Liu, C., Chen, L.-C., Schroff, F., Adam, H., Hua, W., Yuille, A., & Fei-Fei, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00017 · Cited by 1,076 · see citing works
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,009 · see citing works
Time series forecasting of petroleum production using deep LSTM recurrent networks
Sagheer, A., & Kotb, M. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.09.082 · Cited by 975 · see citing works
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972 · see citing works
Siamese Recurrent Architectures for Learning Sentence Similarity
Mueller, J., & Thyagarajan, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10350 · Cited by 935 · see citing works
Stock market's price movement prediction with LSTM neural networks
Nelson, D. M. Q., Pereira, A. C. M., & de Oliveira, R. A. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966019 · Cited by 951 · see citing works
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · see citing works
Miikkulainen, R. P., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-815480-9.00015-3 · Cited by 854 · see citing works
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821 · see citing works
FakeBERT: Fake news detection in social media with a BERT-based deep learning approach
Kaliyar, R. K., Goswami, A., & Narang, P. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10183-2 · Cited by 805 · see citing works
Remaining useful life estimation of engineered systems using vanilla LSTM neural networks
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.05.063 · Cited by 789 · see citing works
ConViT: improving vision transformers with soft convolutional inductive biases
d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., & Sagun, L. (2022) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac9830 · Cited by 782 · see citing works
Bansal, M., Goyal, A., & Choudhary, A. (2022) · Decision Analytics Journal · doi:10.1016/j.dajour.2022.100071 · Cited by 852 · see citing works
Network Traffic Classifier With Convolutional and Recurrent Neural Networks for Internet of Things
López‐Martín, M., Carro, B., Sánchez-Esguevillas, A., & Lloret, J. (2017) · IEEE Access · doi:10.1109/access.2017.2747560 · Cited by 772 · see citing works
Deep Biaffine Attention for Neural Dependency ParsingPreprint
Dozat, T., & Manning, C. D. (2016) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.24 · Cited by 948 · see citing works
Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features
Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, S. W. (2017) · IEEE Access · doi:10.1109/access.2017.2778011 · Cited by 751 · see citing works
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 752 · see citing works
Oh, S. L., Ng, E. Y. K., Tan, R., & Acharya, U. R. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.06.002 · Cited by 733 · see citing works
Long Short Term Memory Recurrent Neural Network Classifier for Intrusion Detection
Kim, J., Kim, J., Le, T.-T.-H., & Kim, H. (2016) · International Conference on Platform Technology and Service · doi:10.1109/platcon.2016.7456805 · Cited by 735 · see citing works
A Deep CNN-LSTM Model for Particulate Matter (PM2.5) Forecasting in Smart Cities
Huang, C., & Kuo, P. (2018) · Sensors · doi:10.3390/s18072220 · Cited by 708 · see citing works
Designing neural networks through neuroevolution
Stanley, K. O., Clune, J., Lehman, J., & Miikkulainen, R. P. (2018) · Nature Machine Intelligence · doi:10.1038/s42256-018-0006-z · Cited by 702 · see citing works
Gao, S., Huang, Y., Zhang, S., Han, J., Wang, G., Zhang, M., & Qingsheng, L. (2020) · Journal of Hydrology · doi:10.1016/j.jhydrol.2020.125188 · Cited by 694 · see citing works
A survey and critique of multiagent deep reinforcement learning
Hernández-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 747 · see citing works
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667 · see citing works
Deep Learning for solar power forecasting — An approach using AutoEncoder and LSTM Neural Networks
Gensler, A., Henze, J., Sick, B., & Raabe, N. (2016) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2016.7844673 · Cited by 666 · see citing works
Deep Learning with Long Short-Term Memory for Time Series Prediction
Hua, Y., Zhao, Z., Li, R., Chen, X., Liu, Z., & Zhang, H. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.2019.1800155 · Cited by 632 · see citing works
Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606 · see citing works
Chen, Y., Sun, X., & Jin, Y. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2953131 · Cited by 575 · see citing works
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J. S., Tian, J., Zhang, Y. S., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 614 · see citing works
A Siamese Long Short-Term Memory Architecture for Human Re-identification
Varior, R. R., Shuai, B., Lu, J., Xu, D., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_9 · Cited by 553 · see citing works
Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network
Yuan, X., Li, L., & Wang, Y. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2902129 · Cited by 550 · see citing works
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Wan, Z., Li, H., He, H., & Prokhorov, D. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2879572 · Cited by 548 · see citing works
A survey on LSTM memristive neural network architectures and applications
Smagulova, K., & James, A. P. (2019) · The European Physical Journal Special Topics · doi:10.1140/epjst/e2019-900046-x · Cited by 546 · see citing works
An optimized model using LSTM network for demand forecasting
Abbasimehr, H., Shabani, M., & Yousefi, M. (2020) · Computers & Industrial Engineering · doi:10.1016/j.cie.2020.106435 · Cited by 539 · see citing works
Ellefsen, A. L., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2018) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2018.11.027 · Cited by 526 · see citing works
Online Multi-Target Tracking Using Recurrent Neural Networks
Milan, A., Rezatofighi, S. H., Dick, A., Reid, I. D., & Schindler, K. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.11194 · Cited by 514 · see citing works
RNN-LSTM: From applications to modeling techniques and beyond—Systematic review
Al-Selwi, S. M., Hassan, M. F., Abdulkadir, S. J., Muneer, A., Sumiea, E. H., Alqushaibi, A., & Ragab, M. G. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102068 · Cited by 553 · see citing works
A solution to the learning dilemma for recurrent networks of spiking neurons
Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj, D., Legenstein, R. A., & Maass, W. (2020) · Nature Communications · doi:10.1038/s41467-020-17236-y · Cited by 514 · see citing works
A Review of Deep Learning Models for Time Series Prediction
Han, Z., Zhao, J., Leung, H., Ma, K. F., & Wang, W. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2923982 · Cited by 509 · see citing works
Xu, Z., & Saleh, J. H. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.107530 · Cited by 521 · see citing works
Nguyen, H. D., Tran, K. P., Thomassey, S., & Hamad, M. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102282 · Cited by 507 · see citing works
Deep Learning for Sensor-based Human Activity Recognition
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2022) · ACM Computing Surveys · doi:10.1145/3447744 · Cited by 690 · see citing works
“A multi-label RNN was created with each RNN cell responding to activity recognition of one occupant.”
BackgroundMethodInfluentialEA-LSTM: Evolutionary attention-based LSTM for time series prediction
Li, Y., Zhu, Z., Kong, D., Han, H., & Zhao, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.05.028 · Cited by 492 · see citing works
Cui, Z., Ke, R., Pu, Z., & Wang, Y. (2020) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102674 · Cited by 490 · see citing works
What to Do Next: Modeling User Behaviors by Time-LSTM
Zhu, Y., Li, H., Liao, Y., Wang, B., Guan, Z., Liu, H., & Cai, D. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/504 · Cited by 489 · see citing works
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. V. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1151 · Cited by 487 · see citing works
Vlachas, P. R., Pathak, J., Hunt, B. R., Sapsis, T. P., Girvan, M., Ott, E., & Koumoutsakos, P. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.02.016 · Cited by 491 · see citing works
A Bidirectional LSTM Prognostics Method Under Multiple Operational Conditions
Huang, C., Huang, H., & Li, Y. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2891463 · Cited by 477 · see citing works
Stock Price Prediction via Discovering Multi-Frequency Trading Patterns
Zhang, L., Aggarwal, C. C., & Qi, G.-J. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098117 · Cited by 476 · see citing works
EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM
Yin, Y., Zheng, X., Hu, B., Zhang, Y., & Cui, X. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106954 · Cited by 470 · see citing works
Long short-term memory for machine remaining life prediction
Zhang, J., Wang, P. E., Yan, R., & Gao, R. X. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.05.011 · Cited by 464 · see citing works
Li, P., Zhang, Z., Xiong, Q., Ding, B., Hou, J., Luo, D., Rong, Y., & Li, S. (2020) · Journal of Power Sources · doi:10.1016/j.jpowsour.2020.228069 · Cited by 459 · see citing works
A deep Recurrent Neural Network based approach for Internet of Things malware threat hunting
HaddadPajouh, H., Dehghantanha, A., Khayami, R., & Choo, K. R. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.03.007 · Cited by 450 · see citing works
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 469 · see citing works
BAT: Deep Learning Methods on Network Intrusion Detection Using NSL-KDD Dataset
Su, T., Sun, H., Zhu, J., Wang, S., & Li, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2972627 · Cited by 436 · see citing works
Predicting process behaviour using deep learning
Evermann, J., Rehse, J.-R., & Fettke, P. (2017) · Decision Support Systems · doi:10.1016/j.dss.2017.04.003 · Cited by 431 · see citing works
A novel spatiotemporal convolutional long short-term neural network for air pollution prediction
Wen, C., Liu, S., Yao, X., Peng, L., Xiang, L., Hu, Y. C., & Chi, T. (2018) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2018.11.086 · Cited by 429 · see citing works
Hate Speech Dataset from a White Supremacy Forum
de Gibert, O., Pérez, N., Pablos, A. G., & Cuadros, M. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5102 · Cited by 428 · see citing works
Overview of the Transformer-based Models for NLP Tasks
Gillioz, A., Casas, J., Mugellini, E., & Khaled, O. A. (2020) · Annals of Computer Science and Information Systems · doi:10.15439/2020f20 · Cited by 447 · see citing works
Fault Diagnosis for Electro-Mechanical Actuators Based on STL-HSTA-GRU and SM
Zhang, X., Li-wei, T., & Chen, J. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3127641 · Cited by 423 · see citing works
Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study
Cañizo, M., Isaac Triguero, A., Conde, Á., & Onieva, E. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.034 · Cited by 422 · see citing works
Anomaly Detection in Automobile Control Network Data with Long Short-Term Memory Networks
Taylor, A., Leblanc, S. P., & Japkowicz, N. (2016) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.20 · Cited by 421 · see citing works
Understanding Neural Networks through Representation ErasurePreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 493 · see citing works
Light Gated Recurrent Units for Speech Recognition
Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2017.2762739 · Cited by 415 · see citing works
Fault diagnosis and remaining useful life estimation of aero engine using LSTM neural network
Yuan, M., Wu, Y., & Li, L. (2016) · IEEE International Conference on Aircraft Utility Systems · doi:10.1109/aus.2016.7748035 · Cited by 410 · see citing works
Why ResNet Works? Residuals Generalize
He, F., Liu, T., & Tao, D. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2966319 · Cited by 409 · see citing works
Cui, Z., Ke, R., Pu, Z., & Wang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02143 · Cited by 406 · see citing works
Rainfall–runoff prediction at multiple timescales with a single Long Short-Term Memory network
Gauch, M., Kratzert, F., Klotz, D., Nearing, G., Lin, J., & Hochreiter, S. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-2045-2021 · Cited by 402 · see citing works
Where to Go Next: A Spatio-Temporal Gated Network for Next POI Recommendation
Zhao, P., Luo, A., Liu, Y., Xu, J., Li, Z., Zhuang, F., Sheng, V. S., & Zhou, X. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3007194 · Cited by 395 · see citing works
Ghimire, S., Deo, R. C., Raj, N., & Mi, J. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.113541 · Cited by 394 · see citing works
Guo, J., Wen, C.-K., Jin, S., & Li, G. Y. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.2968430 · Cited by 392 · see citing works
Deep Learning Models for Cyber Security in IoT Networks
Roopak, M., Tian, G. Y., & Chambers, J. A. (2019) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc.2019.8666588 · Cited by 390 · see citing works
Unsupervised Anomaly Detection With LSTM Neural Networks
Ergen, T., & Kozat, S. S. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2935975 · Cited by 389 · see citing works
Deep learning for load forecasting with smart meter data: Online Adaptive Recurrent Neural Network
Fekri, M. N., Patel, H., Grolinger, K., & Sharma, V. K. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.116177 · Cited by 387 · see citing works
Detecting Cyber Attacks in Industrial Control Systems Using Convolutional Neural Networks
Kravchik, M., & Shabtai, A. (2018) · Workshop on Cyber-Physical Systems Security and PrivaCy (CPS-SPC) · doi:10.1145/3264888.3264896 · Cited by 393 · see citing works
Stock price prediction based on deep neural networks
Yu, P., & Yan, X. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04212-x · Cited by 379 · see citing works
All in One Bad Weather Removal Using Architectural Search
Li, R., Tan, R. T., & Cheong, L.-F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00324 · Cited by 377 · see citing works
A method for state-of-charge estimation of lithium-ion batteries based on PSO-LSTM
Ren, X., Liu, S., Yu, X., & Dong, X. (2021) · Energy · doi:10.1016/j.energy.2021.121236 · Cited by 375 · see citing works
IONet: Learning to Cure the Curse of Drift in Inertial Odometry
Chen, C., Lu, C. X., Markham, A., & Trigoni, N. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12102 · Cited by 374 · see citing works
Optimizing LSTM for time series prediction in Indian stock market
Yadav, A., Jha, C. K., & Sharan, A. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.03.257 · Cited by 368 · see citing works
A hybrid model integrating deep learning with investor sentiment analysis for stock price prediction
Jing, N., Wu, Z., & Wang, H. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115019 · Cited by 391 · see citing works
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Şişman, B., Yamagishi, J., King, S., & Li, H. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 374 · see citing works
Streamflow and rainfall forecasting by two long short-term memory-based models
Ni, L., Wang, D., Singh, V. P., Wu, J., Wang, Y., Tao, Y., & Zhang, J. (2019) · Journal of Hydrology · doi:10.1016/j.jhydrol.2019.124296 · Cited by 361 · see citing works
- Authors
- Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J.
- Affiliations
- University of Applied Sciences and Arts of Southern Switzerland (CH)
- ORCID
- Greff, Klaus
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2017-10
- Volume
- 28
- Issue
- 10
- Pages
- 2222-2232
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W1689711448
- Cited by
- 6,955
- References
- 53
- Citations by year
- 2026: 602 · 2025: 812 · 2024: 810 · 2023: 822 · 2022: 872 · 2021: 881 · 2020: 778 · 2019: 686 · 2018: 383 · 2017: 188 · 2016: 95 · 2015: 25
- FWCI
- 370.42
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Swiss National Science Foundation through the Project Theory and Practice of Reinforcement Learning 2, and through the Project Advanced Reinforcement Learning (138219, 156682)
- European Institute of Innovation and Technology through the Project Neural Dynamics, through the Project NASCENCE and through the Project WAY (FP7-ICT-270247, FP7-ICT-317662, FP7-ICT-288551)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- long short-term memory, recurrent neural networks, hyperparameter optimization, random search, functional analysis of variance, forget gate, speech recognition, handwriting recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
