The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Recurrent nets that time and count
Abstract
The size of the time intervals between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While hidden Markov models tend to ignore this information, recurrent neural networks (RNN) can in principle learn to make use of it. We focus on long short-term memory (LSTM) because it usually outperforms other RNN. Surprisingly, LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes separated by either 50 or 49 discrete time steps, without the help of any short training exemplars. Without external resets or teacher forcing or loss of performance on tasks reported earlier, our LSTM variant also learns to generate very stable sequences of highly nonlinear, precisely timed spikes. This makes LSTM a promising approach for real-world tasks that require to time and count.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Self-citationAdvances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · 9th International Conference on Artificial Neural Networks: ICANN '99 · doi:10.1049/cp:19991218 · Cited by 2,337
Self-citationGradient calculations for dynamic recurrent neural networks: a survey
Pearlmutter, B. A. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410363 · Cited by 582
Experiments on Learning by Back Propagation.
Plaut, D. C., Nowlan, S. J., & Hinton, G. E. (1986) · Cited by 323
Adaptive neural oscillator using continuous-time back-propagation learning
Doya, K., & Yoshizawa, S. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90022-1 · Cited by 129
Recurrent Neural Networks Can Learn to Implement Symbol-Sensitive Counting
Rodriguez, P. F., & Wiles, J. (1997) · Cited by 28
Tsung, F.-S., & Cottrell, G. W. (1994) · Cited by 22
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Self-citationGreff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
Self-citationA Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,591
A Critical Review of Recurrent Neural Networks for Sequence Learning
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 1,996
A review on the long short-term memory model
Van Houdt, G., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 1,991
A State-of-the-Art Survey on Deep Learning Theory and Architectures
Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., Hasan, M., Van Essen, B. C., Awwal, A. A. S., & Asari, V. K. (2019) · Electronics · doi:10.3390/electronics8030292 · Cited by 1,620
DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG
Supratak, A., Dong, H., Wu, C., & Guo, Y. (2017) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2017.2721116 · Cited by 1,376
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958
“There is an extensive body of research on each of these topics, and we will attempt to highlight only the most representative or relevant work since a full treatment is not possible.”
BackgroundLearning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 732
LSTM recurrent networks learn simple context-free and context-sensitive languages
Gers, F. A., & Schmidhuber, E. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963769 · Cited by 722
Self-citationFederated-Learning-Based Anomaly Detection for IoT Security Attacks
Mothukuri, V., Khare, P., Parizi, R. M., Pouriyeh, S., Dehghantanha, A., & Srivastava, G. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3077803 · Cited by 722
An Introductory Review of Deep Learning for Prediction Models With Big Data
Emmert-Streib, F., Yang, Z., Feng, H., Tripathi, S., & Dehmer, M. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00004 · Cited by 604
Transductive LSTM for time-series prediction: An application to weather forecasting
Karevan, Z., & Suykens, J. A. K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.12.030 · Cited by 578
Can deep learning beat numerical weather prediction?
Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L. H., Mozaffari, A., & Stadtler, S. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0097 · Cited by 562
A Siamese Long Short-Term Memory Architecture for Human Re-identification
Varior, R. R., Shuai, B., Lu, J., Xu, D., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_9 · Cited by 524
A Realization-Theoretic Foundation for Fault Diagnosis, with Diagnosability Equivalence Predictions
Bronstein, M. M., Bruna, J., Cohen, T., & Veličković, P. (2021) · arXiv (Cornell University) · doi:10.4230/oasics.dx.2026.1 · Cited by 519
A Review of Deep Learning Models for Time Series Prediction
Han, Z., Zhao, J., Leung, H., Ma, K. F., & Wang, W. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2923982 · Cited by 501
Machine learning for estimation of building energy consumption and performance: a review
Seyedzadeh, S., Rahimian, F. P., Glesk, I., & Roper, M. (2018) · Visualization in Engineering · doi:10.1186/s40327-018-0064-7 · Cited by 482
What to Do Next: Modeling User Behaviors by Time-LSTM
Zhu, Y., Li, H., Liao, Y., Wang, B., Guan, Z., Liu, H., & Cai, D. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/504 · Cited by 468
Li, P., Zhang, Z., Xiong, Q., Ding, B., Hou, J., Luo, D., Rong, Y., & Li, S. (2020) · Journal of Power Sources · doi:10.1016/j.jpowsour.2020.228069 · Cited by 457
A review of deep learning with special emphasis on architectures, applications and recent trends
Sengupta, S., Basak, S., Saikia, P., Paul, S., Tsalavoutis, V., Atiah, F., Ravi, V., & Peters, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105596 · Cited by 454
Stock Price Prediction via Discovering Multi-Frequency Trading Patterns
Zhang, L., Aggarwal, C., & Qi, G.-J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098117 · Cited by 447
Predicting process behaviour using deep learning
Evermann, J., Rehse, J.-R., & Fettke, P. (2017) · Decision Support Systems · doi:10.1016/j.dss.2017.04.003 · Cited by 427
Light Gated Recurrent Units for Speech Recognition
Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2017.2762739 · Cited by 411
LSTM-Based ECG Classification for Continuous Monitoring on Personal Wearable Devices
Saadatnejad, S., Oveisi, M., & Hashemi, M. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2911367 · Cited by 406
Time series analysis and long short-term memory neural network to predict landslide displacement
Yang, B., Yin, K., Lacasse, S., & Liu, Z. (2019) · Landslides · doi:10.1007/s10346-018-01127-x · Cited by 397
LSTM-Based EEG Classification in Motor Imagery Tasks
Wang, P., Jiang, A., Liu, X., Shang, J., & Zhang, L. (2018) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2018.2876129 · Cited by 358
Applying LSTM to Time Series Predictable through Time-Window Approaches
Gers, F. A., Eck, D., & Schmidhuber, J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44668-0_93 · Cited by 334
Self-citationLSTM with sentence representations for document-level sentiment classification
Rao, G., Huang, W., Feng, Z., & Cong, Q. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.04.045 · Cited by 326
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Neil, D., Pfeiffer, M., & Liu, S. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.1610.09513 · Cited by 319
Zhang, Z., Ye, L., Qin, H., Liu, Y., Wang, C., Yu, X., Yin, X., & Li, J. (2019) · Applied Energy · doi:10.1016/j.apenergy.2019.04.047 · Cited by 299
Miao, H., Li, B., Sun, C., & Liu, J. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2900295 · Cited by 293
Multi-output bus travel time prediction with convolutional LSTM neural network
Petersen, N. C., Rodrigues, F., & Pereira, F. C. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.028 · Cited by 288
Working Memory Connections for LSTM
Landi, F., Baraldi, L., Cornia, M., & Cucchiara, R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.08.030 · Cited by 288
Speech Emotion Classification Using Attention-Based LSTM
Xie, Y., Liang, R., Liang, Z., Huang, C., Zou, C., & Schuller, B. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2925934 · Cited by 282
On the Practical Computational Power of Finite Precision RNNs for Language Recognition
Weiss, G., Goldberg, Y., & Yahav, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2117 · Cited by 226
Can recurrent neural networks warp time?
Tallec, C., & Ollivier, Y. (2018) · arXiv · doi:10.48550/arxiv.1804.11188 · Cited by 120
Federated Learning Of Out-Of-Vocabulary Words
Chen, M., Mathews, R., Ouyang, T., & Beaufays, F. (2019) · arXiv · doi:10.48550/arxiv.1903.10635 · Cited by 97
Eom, Y. H., Chung, Y., Park, M., Hong, S. B., & Kim, M. S. (2021) · Energy · doi:10.1016/j.energy.2021.120542 · Cited by 81
A multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61
Time2Vec: Learning a Vector Representation of Time
Kazemi, S. M., Goel, R., Eghbali, S., Ramanan, J., Sahota, J., Thakur, S., Wu, S., Smyth, C., Poupart, P., & Brubaker, M. (2019) · arXiv · doi:10.48550/arxiv.1907.05321 · Cited by 206
In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
Suzuki, K., Kanamura, M., Suga, Y., Mori, H., & Ogata, T. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635954 · Cited by 31
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-1380 · Cited by 10
Metadata
- Authors
- Gers, F. A., & Schmidhuber, J.
- Venue
- Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium
- Publisher
- IEEE
- Published
- 2000
- Pages
- 189-194 vol.3
- OpenAlex
- W2100649405
- Cited by
- 623
- References
- 11
- FWCI
- 2.54
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.