Gated Feedback Recurrent Neural Networks
Open accessAbstract
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LSTM can produce more accurate results than GRU while working with datasets that contain longer sequences.
“although LSTM can produce more accurate results while working with datasets that contain longer sequences.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [111]
Table 2 (p. 6) shows lower test-set error for LSTM than GRU in the stacked and gated-feedback models, though not in the single-layer models. Section 5.2 (p. 7) discusses accuracy as sequence length increases, but compares gated-feedback with stacked architectures, not LSTM with GRU at longer lengths. The claimed longer-sequence qualification is therefore unsupported. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRCNN-based techniques could fully use spatial and temporal information in deepfake movies.
“RCNN-based techniques could fully use spatial and temporal information in deepfake movies”
From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as Chung et al., 2015
The Abstract (p. 1) and Conclusion (p. 8) describe a gated-feedback RNN tested on language modeling and Python program evaluation. A search for RCNNs, spatial and temporal information, video, and deepfakes found no passage addressing the claimed use in deepfake movies. (LLM, conf=0.78)
Experimental — accuracy not yet measuredIn terms of time and space complexity, GRU is more effective than LSTM.
“In terms of time and space complexity, GRU is more effective than LSTM [111]”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [111]
The GRU description (Section 2.1.2, p. 3) explains how its gates and memory differ from LSTM, but does not compare their time or space complexity. The wall-clock results (Section 5.1, p. 6) concern the gated-feedback architecture, not GRU versus LSTM. Searches for computational cost, parameter count, and memory use found no passage establishing the claimed advantage. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 23 · Cited by 583
Research works printed but not linked: 23
- Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. Technical report, arXiv preprint arXiv:1409.0473, 2014. · arXiv
- Bastien, Fr´ ed´ eric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
- Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5 (2):157-166, 1994.
- Bengio, Yoshua, Ducharme, R´ ejean, and Vincent, Pascal. A neural probabilistic language model. In Adv. Neural Inf. Proc. Sys. 13 , pp. 932-938, 2001.
- Cho, Kyunghyun, Van Merri¨ enboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014. · arXiv
- El Hihi, Salah and Bengio, Yoshua. Hierarchical recurrent neural networks for long-term dependencies. In Advances in Neural Information Processing Systems , pp. 493-499. Citeseer, 1995.
- Gers, Felix A., Schmidhuber, J¨ urgen, and Cummins, Fred A. Learning to forget: Continual prediction with LSTM. Neural Computation , 12(10):2451-2471, 2000.
- Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Fr´ ed´ eric, and Bengio, Yoshua. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214 , 2013. · arXiv
- Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013. · arXiv
- Hermans, Michiel and Schrauwen, Benjamin. Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems , pp. 190-198, 2013.
- Hochreiter, Sepp. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut f¨ ur Informatik, Lehrstuhl Prof. Brauer, Technische Universit¨ at M¨ unchen, 1991. URL http://www7. informatik.tu-muenchen.de/˜Ehochreit .
- Hochreiter, Sepp. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems , 6(02):107-116, 1998.
- Hochreiter, Sepp and Schmidhuber, J¨ urgen. Long shortterm memory. Neural Computation , 9(8):1735-1780, 1997.
- Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Koutn´ ık, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, J¨ urgen. A clockwork rnn. In Proceedings of the 31st International Conference on Machine Learning (ICML'14) , 2014.
- Mikolov, Tomas. Statistical Language Models based on Neural Networks . PhD thesis, Brno University of Technology, 2012.
- Mikolov, Tomas, Sutskever, Ilya, Deoras, Anoop, Le, HaiSon, Kombrink, Stefan, and Cernocky, J. Subword language modeling with neural networks. Preprint , 2012.
- Schmidhuber, J¨ urgen. Learning complex, extended sequences using the principle of history compression. Neural Computation , 4(2):234-242, 1992.
- Stollenga, Marijn F, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, J¨ urgen. Deep networks with internal selective attention through feedback connections. In Advances in Neural Information Processing Systems , pp. 3545-3553, 2014.
- Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML'11) , pp. 1017-1024, 2011.
- Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems , pp. 3104-3112, 2014.
- Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615 , 2014. · arXiv
- Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv
- Authors
- Chung, J., Gulcehre, C., Cho, K., & Bengio, Y.
- Affiliations
- Canadian Institute for Advanced Research (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- Pages
- 2067-2075
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1502.02367
- OpenAlex
- W1847088711
- Cited by
- 583
- References
- 23
- Citations by year
- 2025: 34 · 2024: 53 · 2023: 69 · 2022: 59 · 2021: 65 · 2020: 87 · 2019: 83 · 2018: 75 · 2017: 39 · 2016: 15 · 2015: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- Canadian Institute for Advanced Research
- Compute Canada
- Canada Research Chairs
- Samsung
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Environmental Science › Environmental Engineering › Hydrological Forecasting Using AI
- Keywords
- top-down connections, long short-term memory, gated recurrent unit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
