The DOI matches its DataCite record; 19 of the 25 references it cites were found in a registry or an index.
Can recurrent neural networks warp time?
Open accessAbstract
Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a recurrent model formally provide quasi- invariance to general time transformations in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. This result leads to a new way of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Instead, we view it as most closely related to the gating mechanism of traditional RNNs, which is a special case (Theorem 1) and also has a deeper history of connections to SSMs through variable (input-dependent) discretization of Δ (Funahashi and Nakamura 1993; Gu, Dao, et al. 2020; Tallec and Ollivier 2018).”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Tallec and Ollivier 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It also has connections to gating mechanisms of RNNs (Gu, Gulcehre, et al. 2020; Tallec and Ollivier 2018) which we will revisit in Section 3.5.”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Tallec and Ollivier 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · International Conference on Artificial Neural Networks · doi:10.1049/cp:19991218 · Cited by 2,467
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
Exact solutions to the nonlinear dynamics of learning in deep linear neural networksPreprint
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 1,010
Flagged first, then by citations
6 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128, 2016.
- Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pp. 6645-6649. IEEE, 2013.
- Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. In International Conference on Machine Learning, pp. 2034-2042, 2016.
- S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991.
- Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient- based learning. Shape, contour and grouping in computer vision, pp. 823-823, 1999.
- Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cer- nocky. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf), 2012.
- Michael C Mozer. Induction of multiscale temporal structure. In Advances in neural information processing systems, pp. 275-282, 1992.
- Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Cited by
Sorted by citations
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and GaugesPreprint
Bronstein, M. M., Bruna, J., Cohen, T., & Veličković, P. (2021) · arXiv (Cornell University) · doi:10.4230/oasics.dx.2026.1 · Cited by 2,094
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
Hyperbolic Neural NetworksPreprint
Ganea, O.-E., Bécigneul, G., & Hofmann, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09112 · Cited by 347
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space LayersPreprint
Gu, A., Johnson, I., Goel, K., Saab, K. K., Dao, T., Rudra, A., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13985 · Cited by 337
HiPPO: Recurrent Memory with Optimal Polynomial ProjectionsPreprint
Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07669 · Cited by 292
Metadata
- Authors
- Tallec, C., & Ollivier, Y.
- Affiliations
- Université Paris-Sud (FR)
- Meta (United States) (US)
- ORCID
- Tallec, Corentin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1804.11188
- OpenAlex
- W2786406400
- Cited by
- 120
- References
- 25
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 12 · 2023: 13 · 2022: 13 · 2021: 18 · 2020: 13 · 2019: 22 · 2018: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Environmental Science › Environmental Engineering › Hydrological Forecasting Using AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- recurrent neural networks, long short-term memory, gated recurrent unit, time invariance, time transformation, axiomatic method, long-range dependencies, gradient vanishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
