The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
LSTM recurrent networks learn simple context-free and context-sensitive languages
Abstract
Previous work on learning regular languages from exemplary training sequences showed that long short-term memory (LSTM) outperforms traditional recurrent neural networks (RNNs). We demonstrate LSTMs superior performance on context-free language benchmarks for RNNs, and show that it works even better than previous hardwired or highly specialized architectures. To the best of our knowledge, LSTM variants are also the first RNNs to learn a simple context-sensitive language, namely a(n)b(n)c(n).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823
Self-citationGeneralization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 944
Recurrent nets that time and count
Gers, F. A., & Schmidhuber, J. (2000) · Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium · doi:10.1109/ijcnn.2000.861302 · Cited by 623
Self-citationGradient calculations for dynamic recurrent neural networks: a survey
Pearlmutter, B. A. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410363 · Cited by 582
An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories
Williams, R. J., & Peng, J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.4.490 · Cited by 577
Gradient-based learning algorithms for recurrent networks and their computational complexity
Williams, R. J., & Zipser, D. (1995) · Cited by 424
Learning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398
Turing computability with neural nets
Siegelmann, H. T., & Sontag, E. D. (1991) · Applied Mathematics Letters · doi:10.1016/0893-9659(91)90080-f · Cited by 377
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,591
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
A review on the long short-term memory model
Van Houdt, G., Mosquera, C., & Nápoles, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09838-1 · Cited by 1,991
Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks
Segler, M. H. S., Kogej, T., Tyrchan, C., & Waller, M. P. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00512 · Cited by 1,537
An experimental unification of reservoir computing methods
Verstraeten, D., Schrauwen, B., D’Haene, M., & Stroobandt, D. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.003 · Cited by 1,135
Deep learning for short-term traffic flow prediction
Polson, N. G., & Sokolov, V. O. (2017) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2017.02.024 · Cited by 1,015
Sak, H., Senior, A., & Beaufays, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.1128 · Cited by 838
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Dupoux, E., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 737
Deep Learning and Its Applications in Biomedicine
Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., & Xie, Z. (2018) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2017.07.003 · Cited by 624
Performance Evaluation of Deep Neural Networks Applied to Speech Recognition: RNN, LSTM and GRU
Shewalkar, A., Nyavanandi, D., & Ludwig, S. A. (2019) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2019-0006 · Cited by 576
Deep Learning for an Effective Nonorthogonal Multiple Access Scheme
Gui, G., Huang, H., Song, Y., & Sari, H. (2018) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2018.2848294 · Cited by 505
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1108 · Cited by 465
Synthetic Data for Deep Learning
Nikolenko, S. I. (2021) · Springer Optimization and Its Applications · doi:10.1007/978-3-030-75178-4 · Cited by 423
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2188 · Cited by 407
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 371
Advances in De Novo Drug Design: From Conventional to Machine Learning Methods
Mouchlis, V. D., Afantitis, A., Serra, A., Fratello, M., Papadiamantis, A. G., Aidinis, V., Lynch, I., Greco, D., & Melagraki, G. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22041676 · Cited by 353
Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., & Chi, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1054 · Cited by 351
Applying LSTM to Time Series Predictable through Time-Window Approaches
Gers, F. A., Eck, D., & Schmidhuber, J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44668-0_93 · Cited by 334
Self-citationSpoken language understanding using long short-term memory neural networks
Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G., & Shi, Y. (2014) · 2014 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078572 · Cited by 317
Machine Learning-based traffic prediction models for Intelligent Transportation Systems
Boukerche, A., & Wang, J. (2020) · Computer Networks · doi:10.1016/j.comnet.2020.107530 · Cited by 317
Spectral-Spatial Unified Networks for Hyperspectral Image Classification
Xu, Y., Zhang, L., Du, B., & Zhang, F. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2827407 · Cited by 316
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303768966139 · Cited by 289
A survey on the application of recurrent neural networks to statistical language modeling
De Mulder, W., Bethard, S., & Moens, M.-F. (2015) · Computer Speech & Language · doi:10.1016/j.csl.2014.09.005 · Cited by 272
Voice conversion using deep Bidirectional Long Short-Term Memory based Recurrent Neural Networks
Sun, L., Kang, S., Li, K., & Meng, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178896 · Cited by 257
Theoretical Limitations of Self-Attention in Neural Sequence Models
Hahn, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00306 · Cited by 165
Multi-modality Sensor Data Classification with Selective Attention
Zhang, X., Yao, L., Huang, C., Wang, S., Tan, M., Long, G., & Wang, C. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/432 · Cited by 41
Hupkes, D., Veldhoen, S., & Zuidema, W. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11196 · Cited by 14
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
Lakretz, Y., Desbordes, T., Hupkes, D., & Dehaene, S. (2021) · arXiv · doi:10.48550/arxiv.2110.07240 · Cited by 4
Irie, K., Csordás, R., & Schmidhuber, J. (2023) · arXiv · doi:10.48550/arxiv.2310.16076 · Cited by 3
Metadata
- Authors
- Gers, F. A., & Schmidhuber, E.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 2001
- Volume
- 12
- Issue
- 6
- Pages
- 1333-1340
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2121029939
- Cited by
- 722
- References
- 23
- FWCI
- 9.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.