The DOI matches its Crossref record.
Learning to forget: continual prediction with LSTM
Abstract
Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them in an elegant way. 1 Introduction Recurrent neural networks (RNNs) constitute a very powerful class of computational models, capable of ...
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567
Ma, X., Tao, Z., Wang, Y., Yu, H., & Wang, Y. (2015) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2015.03.014 · Cited by 2,191
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,246
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks
Wu, Y., Deng, L., Li, G., Zhu, J., & Shi, L. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00331 · Cited by 1,196
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification
Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Lv, J. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2983860 · Cited by 870
ConViT: improving vision transformers with soft convolutional inductive biases
d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., & Sagun, L. (2022) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac9830 · Cited by 782
Gated Siamese Convolutional Neural Network Architecture for Human Re-identification
Varior, R. R., Haloi, M., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_48 · Cited by 723
Aslam, S., Herodotou, H., Mohsin, S. M., Javaid, N., Ashraf, N., & Aslam, S. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.110992 · Cited by 649
Tsiouris, Κ. Μ., Pezoulas, V. C., Zervakis, M., Konitsiotis, S., Koutsouris, D. D., & Fotiadis, D. I. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.05.019 · Cited by 632
Recurrent nets that time and count
Gers, F. A., & Schmidhuber, J. (2000) · Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium · doi:10.1109/ijcnn.2000.861302 · Cited by 623
Self-citationChen, Y., Sun, X., & Jin, Y. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2953131 · Cited by 573
Recent Advances in Recurrent Neural Networks
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., & Valaee, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01078 · Cited by 562
Fink, O., Wang, Q., Svensén, M., Dersin, P., Lee, W.-J., & Ducoffe, M. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.103678 · Cited by 549
A survey on LSTM memristive neural network architectures and applications
Smagulova, K., & James, A. P. (2019) · The European Physical Journal Special Topics · doi:10.1140/epjst/e2019-900046-x · Cited by 544
A hybrid deep learning model for short-term PV power forecasting
Li, P., Zhou, K., Lu, X., & Yang, S. (2020) · Applied Energy · doi:10.1016/j.apenergy.2019.114216 · Cited by 529
A Siamese Long Short-Term Memory Architecture for Human Re-identification
Varior, R. R., Shuai, B., Lu, J., Xu, D., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_9 · Cited by 524
Artificial Intelligence and Marketing: Pitfalls and Opportunities
De Bruyn, A., Viswanathan, V., Beh, Y. S., Brock, J. K.-U., & Von Wangenheim, F. (2020) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2020.04.007 · Cited by 515
From Feedforward to Recurrent LSTM Neural Networks for Language Modeling
Sundermeyer, M., Ney, H., & Schluter, R. (2015) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2015.2400218 · Cited by 507
Deep neural networks for bot detection
Kudugunta, S., & Ferrara, E. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.019 · Cited by 506
A Review of Deep Learning Models for Time Series Prediction
Han, Z., Zhao, J., Leung, H., Ma, K. F., & Wang, W. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2923982 · Cited by 501
Cui, Z., Ke, R., Pu, Z., & Wang, Y. (2020) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2020.102674 · Cited by 486
Li, P., Zhang, Z., Xiong, Q., Ding, B., Hou, J., Luo, D., Rong, Y., & Li, S. (2020) · Journal of Power Sources · doi:10.1016/j.jpowsour.2020.228069 · Cited by 457
Process‐Guided Deep Learning Predictions of Lake Water Temperature
Read, J. S., Jia, X., Willard, J., Appling, A. P., Zwart, J. A., Oliver, S. K., Karpatne, A., Hansen, G. J. A., Hanson, P. C., Watkins, W., et al. (2019) · Water Resources Research · doi:10.1029/2019wr024922 · Cited by 447
A deep Recurrent Neural Network based approach for Internet of Things malware threat hunting
HaddadPajouh, H., Dehghantanha, A., Khayami, R., & Choo, K.-K. R. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.03.007 · Cited by 445
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
The Challenge of Machine Learning in Space Weather: Nowcasting and Forecasting
Camporeale, E. (2019) · Space Weather · doi:10.1029/2018sw002061 · Cited by 403
Deep learning in bioinformatics: Introduction, application, and perspective in the big data era
Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., & Gao, X. (2019) · Methods · doi:10.1016/j.ymeth.2019.04.008 · Cited by 401
Cui, Z., Ke, R., Pu, Z., & Wang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02143 · Cited by 377
Artificial Neural Networks for Neuroscientists: A Primer
Yang, G. R., & Wang, X.-J. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.005 · Cited by 376
Atitallah, S. B., Driss, M., Boulila, W., & Ghézala, H. B. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100303 · Cited by 375
Hossain, M. S., & Mahmood, H. (2020) · IEEE Access · doi:10.1109/access.2020.3024901 · Cited by 369
Rainfall–runoff prediction at multiple timescales with a single Long Short-Term Memory network
Gauch, M., Kratzert, F., Klotz, D., Nearing, G., Lin, J., & Hochreiter, S. (2021) · Hydrology and Earth System Sciences · doi:10.5194/hess-25-2045-2021 · Cited by 363
Minimal gated unit for recurrent neural networks
Zhou, G.-B., Wu, J., Zhang, C.-L., & Zhou, Z.-H. (2016) · International Journal of Automation and Computing · doi:10.1007/s11633-016-1006-2 · Cited by 357
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning
Sisman, B., Yamagishi, J., King, S., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3038524 · Cited by 357
Sahoo, B. B., Jha, R., Singh, A., & Kumar, D. (2019) · Acta Geophysica · doi:10.1007/s11600-019-00330-1 · Cited by 339
Load Forecasting Techniques for Power System: Research Challenges and Survey
Ahmad, N., Ghadi, Y., Adnan, M., & Ali, M. (2022) · IEEE Access · doi:10.1109/access.2022.3187839 · Cited by 323
Machine Learning-based traffic prediction models for Intelligent Transportation Systems
Boukerche, A., & Wang, J. (2020) · Computer Networks · doi:10.1016/j.comnet.2020.107530 · Cited by 317
Rai, R., & Sahu, C. K. (2020) · IEEE Access · doi:10.1109/access.2020.2987324 · Cited by 315
Toms, B. A., Barnes, E. A., & Ebert‐Uphoff, I. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms002002 · Cited by 311
Deep Irregular Convolutional Residual LSTM for Urban Traffic Passenger Flows Prediction
Du, B., Peng, H., Wang, S., Bhuiyan, M. Z. A., Wang, L., Gong, Q., Liu, L., & Li, J. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2900481 · Cited by 300
A Survey of Data Mining and Deep Learning in Bioinformatics
Lan, K., Wang, D.-T., Fong, S., Liu, L.-S., Wong, K. K. L., & Dey, N. (2018) · Journal of Medical Systems · doi:10.1007/s10916-018-1003-9 · Cited by 288
Network Anomaly Detection Using LSTM Based Autoencoder
Said Elsayed, M., Le-Khac, N.-A., Dev, S., & Jurcut, A. D. (2020) · Proceedings of the 16th ACM Symposium on QoS and Security for Wireless and Mobile Networks · doi:10.1145/3416013.3426457 · Cited by 285
Land Cover Classification via Multitemporal Spatial Data by Deep Recurrent Neural Networks
Ienco, D., Gaetano, R., Dupaquier, C., & Maurel, P. (2017) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2728698 · Cited by 284
Graph Classification using Structural Attention
Lee, J. B., Rossi, R., & Kong, X. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219980 · Cited by 280
A systematic literature review for network intrusion detection system (IDS)
Abdulganiyu, O. H., Ait Tchakoucht, T., & Saheed, Y. K. (2023) · International Journal of Information Security · doi:10.1007/s10207-023-00682-2 · Cited by 279
Machine Learning Based Indoor Localization Using Wi-Fi RSSI Fingerprints: An Overview
Singh, N., Choe, S., & Punmiya, R. (2021) · IEEE Access · doi:10.1109/access.2021.3111083 · Cited by 272
Tree-to-Sequence Attentional Neural Machine Translation
Eriguchi, A., Hashimoto, K., & Tsuruoka, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1078 · Cited by 265
Federated Learning for UAVs-Enabled Wireless Networks: Use Cases, Challenges, and Open Problems
Brik, B., Ksentini, A., & Bouaziz, M. (2020) · IEEE Access · doi:10.1109/access.2020.2981430 · Cited by 258
Uncertainty estimation with deep learning for rainfall–runoff modeling
Klotz, D., Kratzert, F., Gauch, M., Keefe Sampson, A., Brandstetter, J., Klambauer, G., Hochreiter, S., & Nearing, G. (2022) · Hydrology and Earth System Sciences · doi:10.5194/hess-26-1673-2022 · Cited by 256
Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation
Stollenga, M. F., Byeon, W., Liwicki, M., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1506.07452 · Cited by 244
Crop yield prediction integrating genotype and weather variables using deep learning
Shook, J., Gangopadhyay, T., Wu, L., Ganapathysubramanian, B., Sarkar, S., & Singh, A. K. (2021) · PLOS ONE · doi:10.1371/journal.pone.0252402 · Cited by 222
Kratzert, F., Klotz, D., Hochreiter, S., & Nearing, G. S. (2021) · Hydrology and Earth System Sciences · doi:10.5194/hess-25-2685-2021 · Cited by 213
Pythia: AI-assisted Code Completion System
Svyatkovskiy, A., Zhao, Y., Fu, S., & Sundaresan, N. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330699 · Cited by 169
Jung, J.-W., Heo, H.-S., Kim, J.-H., Shim, H.-J., & Yu, H.-J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1982 · Cited by 166
Learning for Personalized Medicine: A Comprehensive Review From a Deep Learning Perspective
Zhang, S., Bamakan, S. M. H., Qu, Q., & Li, S. (2019) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2018.2864254 · Cited by 125
Can recurrent neural networks warp time?
Tallec, C., & Ollivier, Y. (2018) · arXiv · doi:10.48550/arxiv.1804.11188 · Cited by 120
Stock Market Prediction with High Accuracy using Machine Learning Techniques
Bansal, M., Goyal, A., & Choudhary, A. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.12.028 · Cited by 108
Benchmark on a large cohort for sleep-wake classification with machine learning techniques
Palotti, J., Mall, R., Aupetit, M., Rueschman, M., Singh, M., Sathyanarayana, A., Taheri, S., & Fernandez-Luque, L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0126-9 · Cited by 104
Technical and clinical overview of deep learning in radiology
Ueda, D., Shimazaki, A., & Miki, Y. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0795-3 · Cited by 100
A Review on Deep-Learning-Based Cyberbullying Detection
Hasan, M. T., Hossain, M. A. E., Mukta, M. S. H., Akter, A., Ahmed, M., & Islam, S. (2023) · Future Internet · doi:10.3390/fi15050179 · Cited by 103
ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking
Liu, S., Gao, C., Chen, S., Nie, L. Y., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3038681 · Cited by 93
Wang, J., Chen, Y., Gu, Y., Xiao, Y., & Pan, H. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489106 · Cited by 92
Application of deep reinforcement learning on automated stock trading
Chen, L., & Gao, Q. (2019) · 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess47205.2019.9040728 · Cited by 78
Machine Learning Algorithm for Cryptocurrencies Price Prediction
Awotunde, J. B., Ogundokun, R. O., Jimoh, R. G., Misra, S., & Aro, T. O. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-72236-4_17 · Cited by 58
Applications of Metaheuristics in Reservoir Computing Techniques: A Review
Bala, A., Ismail, I., Ibrahim, R., & Sait, S. M. (2018) · IEEE Access · doi:10.1109/access.2018.2873770 · Cited by 53
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
Fung, Y., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., & Sil, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.133 · Cited by 52
Detection of microgrid cyberattacks using network and system management
Karanfil, M., Rebbah, D. E., Debbabi, M., Kassouf, M., Ghafouri, M., Youssef, E.-N. S., & Hanna, A. (2023) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3218934 · Cited by 51
Sub-Seasonal Climate Forecasting via Machine Learning: Challenges, Analysis, and Advances
He, S., Li, X., DelSole, T., Ravikumar, P., & Banerjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16090 · Cited by 51
A Novel Multi-Attention Driven System for Multi-Label Remote Sensing Image Classification
Sumbul, G., & Demir, B. (2019) · IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2019.8898188 · Cited by 44
Trinh, H. D., Fernandez Gambin, A., Giupponi, L., Rossi, M., & Dini, P. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3028197 · Cited by 40
Multi-modality Sensor Data Classification with Selective Attention
Zhang, X., Yao, L., Huang, C., Wang, S., Tan, M., Long, G., & Wang, C. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/432 · Cited by 41
Healthwalks: sensing fine-grained individual health condition via mobility data
Lin, Z., Lyu, S., Cao, H., Xu, F., Wei, Y., Samet, H., & Li, Y. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3432229 · Cited by 18
Topological signatures for fast mobility analysis
Ghosh, A., Rozemberczki, B., Ramamoorthy, S., & Sarkar, R. (2018) · Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems · doi:10.1145/3274895.3274952 · Cited by 12
“Exercise difficulty prediction in online education systems,”
Fang, J., Zhao, W., & Jia, D. (2019) · 2019 International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw.2019.00053 · Cited by 9
Evolving Long Short-Term Memory Networks
Lobo Neto, V. C., Passos, L. A., & Papa, J. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-50417-5_25 · Cited by 9
Multimodal Data Fusion based on the Global Workspace Theory
Bao, C., Fountas, Z., Olugbade, T., & Bianchi-Berthouze, N. (2020) · arXiv · doi:10.48550/arxiv.2001.09485 · Cited by 9
Quantized Reservoir Computing on Edge Devices for Communication Applications
Liu, S., Liu, L., & Yi, Y. (2020) · 2020 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec50012.2020.00068 · Cited by 5
Irie, K., Csordás, R., & Schmidhuber, J. (2023) · arXiv · doi:10.48550/arxiv.2310.16076 · Cited by 3
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
Metadata
- Authors
- Gers, F. A.
- Venue
- 9th International Conference on Artificial Neural Networks: ICANN '99
- Publisher
- IEE
- Published
- 1999
- Volume
- 1999
- Pages
- 850-855
- OpenAlex
- W2116261113
- Cited by
- 2,337
- References
- 0
- FWCI
- 0.57
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.