The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Learning long-term dependencies in NARX recurrent neural networks
Abstract
It has previously been shown that gradient-descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies, i.e. those problems for which the desired output depends on inputs presented at times far in the past. We show that the long-term dependencies problem is lessened for a class of architectures called nonlinear autoregressive models with exogenous (NARX) recurrent neural networks, which have powerful representational capabilities. We have previously reported that gradient descent learning can be more effective in NARX networks than in recurrent neural network architectures that have "hidden states" on problems including grammatical inference and nonlinear system identification. Typically, the network converges much faster and generalizes better than other networks. The results in this paper are consistent with this phenomenon. We present some experimental results which show that NARX networks can often retain information for two to three times as long as conventional recurrent neural networks. We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on long-term dependency problems. We also describe in detail some of the assumptions regarding what it means to latch information robustly and suggest possible ways to loosen these assumptions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
System Identification: Theory for the User
Lennart Ljung, A. (1987) · Cited by 17,690
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
Kailath, T. (1980) · Cited by 6,215
Input-output parametric models for non-linear systems Part I: deterministic non-linear systems
LEONTARITIS, I. J., & BILLINGS, S. A. (1985) · International Journal of Control · doi:10.1080/0020718508961129 · Cited by 1,147
Input-output parametric models for non-linear systems Part II: stochastic non-linear systems
LEONTARITIS, I. J., & BILLINGS, S. A. (1985) · International Journal of Control · doi:10.1080/0020718508961130 · Cited by 1,065
Non-linear system identification using neural networks
CHEN, S., BILLINGS, S. A., & GRANT, P. M. (1990) · International Journal of Control · doi:10.1080/00207179008934126 · Cited by 1,043
Cited by
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Maier, H. R., & Dandy, G. C. (2000) · Environmental Modelling & Software · doi:10.1016/s1364-8152(99)00007-9 · Cited by 2,377
Electricity price forecasting: A review of the state-of-the-art with a look into the future
Weron, R. (2014) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.08.008 · Cited by 1,646
A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
Qin, Y., Song, D., Chen, H., Cheng, W., Jiang, G., & Cottrell, G. W. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/366 · Cited by 1,585
Adaptive Computation and Machine Learning
(2012) · Boosting · doi:10.7551/mitpress/8291.003.0026 · Cited by 1,402
A survey on river water quality modelling using artificial intelligence models: 2000–2020
Tiyasha, Tung, T. M., & Yaseen, Z. M. (2020) · Journal of Hydrology · doi:10.1016/j.jhydrol.2020.124670 · Cited by 650
A comprehensive review of deep learning applications in hydrology and water resources
Sit, M., Demiray, B. Z., Xiang, Z., Ewing, G. J., Sermet, Y., & Demir, I. (2020) · Water Science and Technology · doi:10.2166/wst.2020.369 · Cited by 618
Basicvsr++: improving video super-resolution with enhanced propagation and alignment
Chan, K. C. K., Zhou, S., Xu, X., & Loy, C. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00588 · Cited by 604
Chataut, R., & Akl, R. (2020) · Sensors · doi:10.3390/s20102753 · Cited by 556
GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction
Liang, Y., Ke, S., Zhang, J., Yi, X., & Zheng, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/476 · Cited by 510
Exploiting the past and the future in protein secondary structure prediction
Baldi, P., Brunak, S., Frasconi, P., Soda, G., & Pollastri, G. (1999) · Bioinformatics · doi:10.1093/bioinformatics/15.11.937 · Cited by 497
Computational capabilities of recurrent NARX neural networks
Siegelmann, H. T., Horne, B. G., & Giles, C. L. (1997) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/3477.558801 · Cited by 455
Self-citationLong-term time series prediction with the NARX network: An empirical evaluation
Menezes, J. M. P., & Barreto, G. A. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.01.030 · Cited by 446
Deep Learning for Stock Market Prediction
Nabipour, M., Nayyeri, P., Jabani, H., Mosavi, A., Salwana, E., & S., S. (2020) · Entropy · doi:10.3390/e22080840 · Cited by 442
A comprehensive review of digital twin — part 1: modeling and twinning enabling technologies
Thelen, A., Zhang, X., Fink, O., Lu, Y., Ghosh, S., Youn, B. D., Todd, M. D., Mahadevan, S., Hu, C., & Hu, Z. (2022) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-022-03425-4 · Cited by 441
SICK, B. (2002) · Mechanical Systems and Signal Processing · doi:10.1006/mssp.2001.1460 · Cited by 439
Hierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386
Sharifzadeh, M., Sikinioti-Lock, A., & Shah, N. (2019) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.03.040 · Cited by 381
A new class of wavelet networks for nonlinear system identification
Billings, S. A., & Wei, H.-L. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.849842 · Cited by 368
Wunsch, A., Liesch, T., & Broda, S. (2021) · Hydrology and Earth System Sciences · doi:10.5194/hess-25-1671-2021 · Cited by 353
$\mathtt {Deepr}$: A Convolutional Net for Medical Records
Nguyen, P., Tran, T., Wickramasinghe, N., & Venkatesh, S. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2633963 · Cited by 352
Applying LSTM to Time Series Predictable through Time-Window Approaches
Gers, F. A., Eck, D., & Schmidhuber, J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44668-0_93 · Cited by 334
DiPietro, R., & Hager, G. D. (2020) · Handbook of Medical Image Computing and Computer Assisted Intervention · doi:10.1016/b978-0-12-816176-0.00026-0 · Cited by 293
Manoharan, A., Begam, K. M., Aparow, V. R., & Sooriamoorthy, D. (2022) · Journal of Energy Storage · doi:10.1016/j.est.2022.105384 · Cited by 293
Recurrent and Dynamic Models for Predicting Streaming Video Quality of Experience
Bampis, C. G., Li, Z., Katsavounidis, I., & Bovik, A. C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2815842 · Cited by 84
Wang, Q., Gu, H., Ye, M., Wei, M., & Xu, X. (2021) · IEEE Access · doi:10.1109/access.2021.3086507 · Cited by 52
Tandem decoding of children's speech for keyword detection in a child-robot interaction scenario
Wöllmer, M., Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998386 · Cited by 18
Metadata
- Authors
- Lin, T., Horne, B. G., Tino, P., & Giles, C. L.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1996-11
- Volume
- 7
- Issue
- 6
- Pages
- 1329-1338
- ISSN
- 1045-9227 (ISSN-L)
- 1941-0093
- OpenAlex
- W2103452139
- Cited by
- 784
- References
- 26
- FWCI
- 6.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.