The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Learning long-term dependencies in NARX recurrent neural networks
Abstract
It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
System Identification: Theory for the User
Lennart Ljung, A. (1987) · Cited by 17,690
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
Kailath, T. (1980) · Cited by 6,215
Input-output parametric models for non-linear systems Part I: deterministic non-linear systems
LEONTARITIS, I. J., & BILLINGS, S. A. (1985) · International Journal of Control · doi:10.1080/0020718508961129 · Cited by 1,147
Input-output parametric models for non-linear systems Part II: stochastic non-linear systems
LEONTARITIS, I. J., & BILLINGS, S. A. (1985) · International Journal of Control · doi:10.1080/0020718508961130 · Cited by 1,065
Non-linear system identification using neural networks
CHEN, S., BILLINGS, S. A., & GRANT, P. M. (1990) · International Journal of Control · doi:10.1080/00207179008934126 · Cited by 1,043
On the Computational Power of Neural Nets
Siegelmann, H. T., & Sontag, E. D. (1995) · Journal of Computer and System Sciences · doi:10.1006/jcss.1995.1013 · Cited by 692
Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks
Puskorius, G. V., & Feldkamp, L. A. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279191 · Cited by 528
Computational capabilities of recurrent NARX neural networks
Siegelmann, H. T., Horne, B. G., & Giles, C. L. (1997) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/3477.558801 · Cited by 455
Self-citationGradient-based learning algorithms for recurrent networks and their computational complexity
Williams, R. J., & Zipser, D. (1995) · Cited by 424
Learning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 325
The gamma model—A new neural model for temporal processing
Vries, B. D., & Principe, J. C. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80035-8 · Cited by 254
FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling
Back, A. D., & Tsoi, A. C. (1991) · Neural Computation · doi:10.1162/neco.1991.3.3.375 · Cited by 210
Local Feedback Multilayered Networks
Frasconi, P., Gori, M., & Soda, G. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.120 · Cited by 189
Comparison of four neural net learning methods for dynamic system identification
Qin, S.-Z., Su, H.-T., & McAvoy, T. J. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.105425 · Cited by 184
Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms
Nerrand, O., Roussel-Ragot, P., Personnaz, L., Dreyfus, G., & Marcos, S. (1993) · Neural Computation · doi:10.1162/neco.1993.5.2.165 · Cited by 183
Unified integration of explicit knowledge and learning by example in recurrent networks
Frasconi, P., Gori, M., Maggini, M., & Soda, G. (1995) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.382304 · Cited by 82
A structure by which a recurrent neural network can approximate a nonlinear dynamic system
Seidl, D. R., & Lorenz, R. D. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.155422 · Cited by 51
Identification of Chemical Processes using Recurrent Networks
Su, H.-T., & McAvoy, T. J. (1991) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.1991.4791818 · Cited by 43
Learning Unambiguous Reduced Sequence Descriptions
Schmidhuber, J. (1991) · Cited by 33
Nonlinear prediction of speech signals using memory neuron networks
Poddar, P., & Unnikrishnan, K. P. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1991.239502 · Cited by 22
Cited by
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Maier, H. R., & Dandy, G. C. (2000) · Environmental Modelling & Software · doi:10.1016/s1364-8152(99)00007-9 · Cited by 2,377
Electricity price forecasting: A review of the state-of-the-art with a look into the future
Weron, R. (2014) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.08.008 · Cited by 1,646
Metadata
- Authors
- Lin, T., Horne, B. G., Tino, P., & Giles, C. L.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1996-11
- Volume
- 7
- Issue
- 6
- Pages
- 1329-1338
- ISSN
- 1045-9227 (ISSN-L)
- 1941-0093
- OpenAlex
- W2103452139
- Cited by
- 784
- References
- 26
- FWCI
- 6.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.