On the difficulty of training recurrent neural networks
Open accessAbstract
There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recurrent models are hard to correctly train.
“and are hard to correctly train [21].”
From Temporal Convolutional Networks for Action Segmentation and Detection (2017) · cites this paper as [21]
RNNs with tanh or sigmoid non-linearities suffer from vanishing gradients and are poor at capturing long temporal dependencies essential for anticipation.
“RNNs with this choice of f suffer from a well-studied problem of vanishing gradients [29], and hence are poor at capturing long temporal dependencies which are essential for anticipation.”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [29]
Jain et al. proposed an Autoregressive Input-Output Hidden Markov Model which fuses sensory streams through a linear transformation of features and performs better than feature concatenation.
“They proposed an Autoregressive InputOutput Hidden Markov Model (AIO-HMM) which fuses sensory streams through a linear transformation of features and it performs better than feature concatenation [27].”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [27]
The Introduction (p. 1) describes recurrent neural networks, and the experiments (pp. 8, 11–12) concern their training and performance. Searches for AIO-HMM, sensory-stream fusion, and feature concatenation found no passage addressing the attributed model or comparison in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredAnticipating driving maneuvers several seconds in advance enables assistive cars to alert drivers before they make a dangerous maneuver.
“In another application, anticipating driving maneuvers several seconds in advance [19, 27, 38, 36] enables assistive cars to alert drivers before they make a dangerous maneuver.”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [27]
The passages from §2.2 (pp. 3–4) discuss recurrent-network dynamics, not driving. Searches for maneuver anticipation, driver alerts, and vehicle prediction found no passage in our copy addressing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredPrevious works treat anticipation as a recognition problem and train discriminative classifiers on the complete temporal context.
“Previous works treat anticipation as a recognition problem [21, 27, 31] and train discriminative classifiers (such as SVM or CRF) on the complete temporal context.”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [27]
The Temporal Order problem (p. 7) describes an RNN classifying symbol order at the end of a sequence, not prior work treating anticipation as recognition. The Natural problems section (p. 8) discusses music and character prediction; searches for anticipation, SVM, and CRF found no passage supporting the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Both F-RNN and AIO-HMM perform significantly better than previous work on maneuver anticipation by Morris et al. [27].
“Additionally, both F-RNN and AIOHMM perform significantly better than previous work on maneuver anticipation by Morris et al. [27] (Fig. 7a).”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [27]
The Natural problems section (p. 8) reports results for music prediction and character-level language modelling, not maneuver anticipation. Searches for Morris, F-RNN, and AIO-HMM found no passage in our copy identifying Morris et al. as prior work on maneuver anticipation or comparing those models with it. (LLM, conf=0.77)
Experimental — accuracy not yet measuredThe FBPConvNet training clips the computed gradients to a fixed range to prevent the divergence of the cost function.
“We clip the computed gradients to a fixed range to prevent the divergence of the cost function [27], [46].”
From Deep Convolutional Neural Network for Inverse Problems in Imaging (2017) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811 · see citing works
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956 · see citing works
Optimization and applications of echo state networks with leaky- integrator neurons
Jaeger, H., Lukoševičius, M., Popovici, D., & Siewert, U. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.016 · Cited by 948 · see citing works
Flagged first, then by citations
Research works printed but not linked: 19
- Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. Submited to Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
- Bengio, Y., Frasconi, P., and Simard, P. (1993). The problem of learning long-term dependencies in recurrent networks. pages 1183-1195, San Francisco. IEEE Press. (invited paper).
- Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5 (2), 157-166.
- Bengio, Y., Boulanger-Lewandowski, N., and Pascanu, R. (2012). Advances in optimizing recurrent networks. Technical Report arXiv:1212.0901, U. Montreal. · arXiv
- Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., WardeFarley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) . Oral Presentation.
- Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in highdimensional sequences: Application to polyphonic music generation and transcription. In Proceedings of the Twenty-nine International Conference on Machine Learning (ICML'12) . ACM.
- Doya, K. (1993). Bifurcations of recurrent neural networks in gradient descent learning. IEEE Transactions on Neural Networks , 1 , 75-80.
- Doya, K. and Yoshizawa, S. (1991). Adaptive synchronization of neural and physical oscillators. In J. E. Moody, S. J. Hanson, and R. Lippmann, editors, NIPS , pages 109-116. Morgan Kaufmann.
- Duchi, J. C., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12 , 2121-2159.
- Elman, J. (1990). Finding structure in time. Cognitive Science , 14 (2), 179-211.
- Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., and Schmidhuber, J. (2009). A Novel Connectionist System for Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 31 (5), 855-868.
- Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimization. In Proc. ICML'2011 . ACM.
- Mikolov, T. (2012). Statistical Language Models based on Neural Networks . Ph.D. thesis, Brno University of Technology.
- Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Cernocky, J. (2011). Empirical evaluation and combination of advanced language modeling techniques. In Proc. 12th annual conference of the international speech communication association (INTERSPEECH 2011) .
- Mikolov, T., Sutskever, I., Deoras, A., Le, H.-S., Kombrink, S., and Cernocky, J. (2012). Subword language modeling with neural networks. preprint (http://www.fit.vutbr.cz/ imikolov/rnnlm/char.pdf).
- Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by backpropagating errors. Nature , 323 (6088), 533-536.
- Strogatz, S. (1994). Nonlinear Dynamics And Chaos: With Applications To Physics, Biology, Chemistry, And Engineering (Studies in Nonlinearity) . Studies in nonlinearity. Perseus Books Group, 1 edition.
- Sutskever, I., Martens, J., and Hinton, G. (2011). Generating text with recurrent neural networks. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11) , ICML '11, pages 1017-1024, New York, NY, USA. ACM.
- Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market model. Neural Networks , 1 (4), 339-356.
Cited by
Sorted by citations
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870 · see citing works
“Adjustable Gradient Clipping Gradient clipping is a technique that is often used in training recurrent neural networks [17].”
MethodFundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680 · see citing works
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · see citing works
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342 · see citing works
“, 2012); (3) RNNs are considered hard to train and parallelize which led the researchers to avoid using them for computational reasons (Pascanu et al., 2013).”
BackgroundConvolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · see citing works
“1 (Pascanu et al., 2013).”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“We use global norm gradient clipping (Pascanu et al. (2012)) with a value of 1.0 for all models.”
MethodInfluentialDeep Convolutional Neural Network for Inverse Problems in Imaging
Jin, K. H., McCann, M. T., Froustey, E., & Unser, M. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2713099 · Cited by 2,529 · see citing works
“to prevent the divergence of the cost function [29], [61].”
BackgroundConditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · see citing works
“Therefore, it does not suffer from the infamous vanishing and exploding gradient problem inherent to temporarily deep RNNs [3, 24].”
BackgroundInfluentialTemporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · see citing works
“typically too short to capture long-range temporal patterns; segmental models [25, 16, 24], which typically condition the current action class on the previous segment, thus ignoring long-range latent dependencies; and recurrent models [28, 10], which empirically can have a limited span of attention [28] and are hard t…”
BackgroundA Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security
Al-Garadi, M. A., Mohamed, A., Al-Ali, A. K., Du, X., Ali, I., & Guizani, M. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2988293 · Cited by 1,392 · see citing works
DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG
Supratak, A., Dong, H., Wu, C., & Guo, Y. (2017) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2017.2721116 · Cited by 1,376 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
Siamese Recurrent Architectures for Learning Sentence Similarity
Mueller, J., & Thyagarajan, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10350 · Cited by 935 · see citing works
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972 · see citing works
Long Short-Term Memory Networks for Accurate State-of-Charge Estimation of Li-ion Batteries
Chemali, E., Kollmeyer, P. J., Preindl, M., Ahmed, R., & Emadi, A. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2787586 · Cited by 906 · see citing works
Remaining useful life estimation of engineered systems using vanilla LSTM neural networks
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.05.063 · Cited by 789 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · see citing works
Zheng, H., Yuan, J., & Chen, L. (2017) · Energies · doi:10.3390/en10081168 · Cited by 635 · see citing works
BEHRT: Transformer for Electronic Health Records
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., & Salimi-Khorshidi, G. (2020) · Scientific Reports · doi:10.1038/s41598-020-62922-y · Cited by 595 · see citing works
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605 · see citing works
“A neural approach to automated essay scoring,”
Taghipour, K., & Ng, H. T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1193 · Cited by 537 · see citing works
Vidal, C., Malysz, P., Kollmeyer, P., & Emadi, A. (2020) · IEEE Access · doi:10.1109/access.2020.2980961 · Cited by 512 · see citing works
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
Predicting healthcare trajectories from medical records: a deep learning approach
Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.001 · Cited by 447 · see citing works
Hierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386 · see citing works
Self-citationDeep Learning for Time Series Forecasting: Tutorial and Literature Survey
Benidis, K., Rangapuram, S. S., Flunkert, V., Wang, Y., Maddix, D., Turkmen, C., Gasthaus, J., Bohlke-Schneider, M., Salinas, D., Stella, L., et al. (2023) · ACM Computing Surveys · doi:10.1145/3533382 · Cited by 382 · see citing works
Gated-Attention Readers for Text Comprehension
Dhingra, B., Liu, H., Yang, Z., Cohen, W., & Salakhutdinov, R. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1168 · Cited by 366 · see citing works
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354 · see citing works
Learning Text Similarity with Siamese Recurrent Networks
Neculoiu, P., Versteegh, M., & Rotaru, M. (2016) · Proceedings of the 1st Workshop on Representation Learning for NLP · doi:10.18653/v1/w16-1617 · Cited by 341 · see citing works
“In this paper we take the latter approach for the flexibility it affords us in dealing with out-of-vocabulary words.”
BackgroundSong, H. F., Yang, G. R., & Wang, X.-J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004792 · Cited by 281 · see citing works
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332 · see citing works
Deep learning goes to school: toward a relational understanding of AI in education
Perrotta, C., & Selwyn, N. (2020) · Learning, Media and Technology · doi:10.1080/17439884.2020.1686017 · Cited by 299 · see citing works
DiPietro, R., & Hager, G. D. (2020) · Handbook of Medical Image Computing and Computer Assisted Intervention · doi:10.1016/b978-0-12-816176-0.00026-0 · Cited by 293 · see citing works
Controllable Abstractive Summarization
Fan, A., Grangier, D., & Auli, M. (2018) · Proceedings of the 2nd Workshop on Neural Machine Translation and Generation · doi:10.18653/v1/w18-2706 · Cited by 281 · see citing works
“1 (Pascanu et al., 2013), momentum 0.”
MethodInfluentialRecurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487478 · Cited by 271 · see citing works
“RNNs with this choice of f suffer from a well-studied problem of vanishing gradients [29], and hence are poor at capturing long temporal dependencies which are essential for anticipation.”
BackgroundMethodUnsupervised Pretraining for Sequence to Sequence Learning
Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251 · see citing works
Cloze-driven Pretraining of Self-attention Networks
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1539 · Cited by 215 · see citing works
Boukerche, A., Tao, Y., & Sun, P. (2020) · Computer Networks · doi:10.1016/j.comnet.2020.107484 · Cited by 215 · see citing works
Learning to Generate Product Reviews from Attributes
Dong, L., Huang, S., Wei, F., Lapata, M., Zhou, M., & Xu, K. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1059 · Cited by 190 · see citing works
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194 · see citing works
Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy
Yeung, M., Sala, E., Schönlieb, C.-B., & Rundo, L. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104815 · Cited by 186 · see citing works
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Fang, K., Toshev, A., Fei-Fei, L., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00063 · Cited by 178 · see citing works
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.233 · Cited by 179 · see citing works
Zhao, J., Feng, X., Wang, J., Lian, Y., Ouyang, M., & Burke, A. F. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121949 · Cited by 152 · see citing works
Automated Crowdturfing Attacks and Defenses in Online Review Systems
Yao, Y., Viswanath, B., Cryan, J., Zheng, H., & Zhao, B. Y. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133990 · Cited by 150 · see citing works
Personalized Transformer for Explainable Recommendation
Li, L., Zhang, Y., & Chen, L. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.383 · Cited by 124 · see citing works
Intent Prediction of Pedestrians via Motion Trajectories Using Stacked Recurrent Neural Networks
Saleh, K., Hossny, M., & Nahavandi, S. (2018) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2018.2873901 · Cited by 103 · see citing works
Factorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · see citing works
“Learning long-range dependencies with Recurrent Neural Networks (RNN) is challenging due to the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013).”
BackgroundA multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61 · see citing works
Deep Reinforcement Learning for Syntactic Error Repair in Student Programs
Gupta, R., Kanade, A., & Shevade, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301930 · Cited by 54 · see citing works
Audio and face video emotion recognition in the wild using deep neural networks and small datasets
Ding, W., Xu, M., Huang, D., Lin, W., Dong, M., Yu, X., & Li, H. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997637 · Cited by 51 · see citing works
Liu, K., Kang, L., & Xie, D. (2023) · Batteries · doi:10.3390/batteries9020094 · Cited by 45 · see citing works
Kreutzer, J., Uyheng, J., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1165 · Cited by 38 · see citing works
Capsulenet-Based Spatial–Spectral Classifier for Hyperspectral Images
Arun, P. V., Buddhiraju, K. M., & Porwal, A. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2913097 · Cited by 37 · see citing works
Gesture recognition with the linear optical sensor and recurrent neural networks
Czuszynski, K., Ruminski, J., & Kwasniewska, A. (2018) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2834968 · Cited by 36 · see citing works
DTR-HAR: deep temporal residual representation for human activity recognition
Basly, H., Ouarda, W., Sayadi, F. E., Ouni, B., & Alimi, A. M. (2022) · The Visual Computer · doi:10.1007/s00371-021-02064-y · Cited by 36 · see citing works
Multimodal Recurrent Neural Networks With Information Transfer Layers for Indoor Scene Labeling
Abdulnabi, A. H., Shuai, B., Zuo, Z., Chau, L.-P., & Wang, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2774007 · Cited by 33 · see citing works
Gupta, S., S., R., K., R., & Verma, R. (2020) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2020.0110941 · Cited by 26 · see citing works
Stylianou, N., Razis, G., Goulis, D. G., & Vlahavas, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101949 · Cited by 22 · see citing works
Tiftikci, M., Özgür, A., He, Y., & Hur, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3195-5 · Cited by 23 · see citing works
Pelka, O., Friedrich, C. M., Nensa, F., Mönninghoff, C., Bloch, L., Jöckel, K.-H., Schramm, S., Sanchez Hoffmann, S., Winkler, A., Weimar, C., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236868 · Cited by 22 · see citing works
“The effective characteristic of LSTM is the ability to accumulate state information, as information of every new input is accumulated onto previous input [69, 70].”
BackgroundRWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17 · see citing works
“, 2019), gradient clipping (Pascanu et al., 2012), and adding constraints (Kanai et al.”
BackgroundFixed complexity tiny reservoir heterogeneous network for on-line ECG learning of anomalies
Abdennadher, N., Pau, D., & Bruna, A. (2021) · 2021 IEEE 10th Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce53005.2021.9622022 · Cited by 9 · see citing works
Baz, M. (2024) · Mathematics · doi:10.3390/math12213320 · Cited by 1 · see citing works
PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Woo, S., Kim, H., Shim, S., Jo, M., Jeong, H., Lee, J., Kim, J., Lee, S., Ryu, J., Park, B., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12029
“There were no explicit regularization setups, including gradient clipping (Pascanu et al., 2013) or dropout (Srivastava et al., 2014).”
- Authors
- Pascanu, R., Mikolov, T., & Bengio, Y.
- Affiliations
- Université de Montréal (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1211.5063
- OpenAlex
- W1815076433
- Cited by
- 4,310
- References
- 4
- Citations by year
- 2026: 15 · 2025: 184 · 2024: 358 · 2023: 413 · 2022: 408 · 2021: 592 · 2020: 621 · 2019: 673 · 2018: 459 · 2017: 309 · 2016: 157 · 2015: 67 · 2014: 39 · 2013: 12 · 2012: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- recurrent neural networks, gradient vanishing, gradient explosion, dynamical systems analysis, training difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
