The DOI matches its Crossref record.
The problem of learning long-term dependencies in recurrent networks
Abstract
The authors seek to train recurrent neural networks in order to map input sequences to output sequences, for applications in sequence recognition or production. Results are presented showing that learning long-term dependencies in such recurrent networks using gradient descent is a very difficult task. It is shown how this difficulty arises when robustly latching bits of information with certain attractors. The derivatives of the output at time t with respect to the unit activations at time zero tend rapidly to zero as t increases for most input values. In such a situation, simple gradient descent techniques appear inappropriate. The consideration of alternative optimization methods and architectures is suggested.>
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Self-citationA comprehensive review of deep learning applications in hydrology and water resources
Sit, M., Demiray, B. Z., Xiang, Z., Ewing, G. J., Sermet, Y., & Demir, I. (2020) · Water Science and Technology · doi:10.2166/wst.2020.369 · Cited by 618
Cognition and control in schizophrenia: a computational model of dopamine and prefrontal function
Braver, T. S., Barch, D. M., & Cohen, J. D. (1999) · Biological Psychiatry · doi:10.1016/s0006-3223(99)00116-x · Cited by 507
Extended Dissipative State Estimation for Markov Jump Neural Networks With Unreliable Links
Shen, H., Zhu, Y., Zhang, L., & Park, J. H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2511196 · Cited by 489
Metadata
- Authors
- Bengio, Y., Frasconi, P., & Simard, P.
- Venue
- IEEE International Conference on Neural Networks
- Publisher
- IEEE
- Published
- 2002-12-30
- Pages
- 1183-1188
- ISSN
- 1098-7576 (ISSN-L)
- OpenAlex
- W2133656308
- Cited by
- 241
- References
- 0
- FWCI
- 1.63
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The problem of learning long-term dependencies in recurrent networks. https://app.cochat.ai/papers/10.1109/icnn.1993.298725
- The paper
- Bengio, Y., Frasconi, P., & Simard, P. (2002). The problem of learning long-term dependencies in recurrent networks. IEEE International Conference on Neural Networks. https://doi.org/10.1109/icnn.1993.298725
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.