The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
New results on recurrent network training: unifying the algorithms and accelerating convergence
Abstract
How to efficiently train recurrent networks remains a challenging and active research topic. Most of the proposed training approaches are based on computational ways to efficiently obtain the gradient of the error function, and can be generally grouped into five major groups. In this study we present a derivation that unifies these approaches. We demonstrate that the approaches are only five different ways of solving a particular matrix equation. The second goal of this paper is develop a new algorithm based on the insights gained from the novel formulation. The new algorithm, which is based on approximating the error gradient, has lower computational complexity in computing the weight update than the competing techniques for most typical problems. In addition, it reaches the error minimum in a much smaller number of iterations. A desirable characteristic of recurrent network training algorithms is to be able to update the weights in an on-line fashion. We have also developed an on-line version of the proposed algorithm, that is based on updating the error gradient approximation in a recursive manner.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Neural Networks: A Comprehensive Foundation
Haykin, S. (1998) · Cited by 30,070
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Horn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
Neural networks: A comprehensive foundation
Tadeusiewicz, R. (1995) · Control Engineering Practice · doi:10.1016/0967-0661(95)90080-2 · Cited by 7,218
Applied Optimal Control: Optimization, Estimation, and Control
Bryson, A. E., & Ho, Y.-C. (2018) · Routledge eBooks · doi:10.1201/9781315137667 · Cited by 6,015
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
Information processing using a single dynamical node as complex system
Appeltant, L., Soriano, M. C., Van der Sande, G., Danckaert, J., Massar, S., Dambre, J., Schrauwen, B., Mirasso, C. R., & Fischer, I. (2011) · Nature Communications · doi:10.1038/ncomms1476 · Cited by 1,716
An experimental unification of reservoir computing methods
Verstraeten, D., Schrauwen, B., D’Haene, M., & Stroobandt, D. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.003 · Cited by 1,135
Generating Coherent Patterns of Activity from Chaotic Neural Networks
Sussillo, D., & Abbott, L. F. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.018 · Cited by 1,092
Reservoir computing using dynamic memristors for temporal information processing
Du, C., Cai, F., Zidan, M. A., Ma, W., Lee, S. H., & Lu, W. D. (2017) · Nature Communications · doi:10.1038/s41467-017-02337-y · Cited by 1,015
Optoelectronic Reservoir Computing
Paquot, Y., Duport, F., Smerieri, A., Dambre, J., Schrauwen, B., Haelterman, M., & Massar, S. (2012) · Scientific Reports · doi:10.1038/srep00287 · Cited by 783
Minimum Complexity Echo State Network
Rodan, A., & Tino, P. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2089641 · Cited by 710
Federated learning for 6G communications: Challenges, methods, and future directions
Liu, Y., Yuan, X., Xiong, Z., Kang, J., Wang, X., & Niyato, D. (2020) · China Communications · doi:10.23919/jcc.2020.09.009 · Cited by 464
“…Level : At the algorithm level, achieving communication-efficient FL can reduce the communication rounds of training a model by accelerating convergence [25] and reduce the communication cost of each round by using gradient compression techniques [10] (e.g., sparsification, quantization, etc.).”
MethodThe general inefficiency of batch training for gradient descent learning
Wilson, D. R., & Martinez, T. R. (2003) · Neural Networks · doi:10.1016/s0893-6080(03)00138-2 · Cited by 418
Lukoševičius, M., Jaeger, H., & Schrauwen, B. (2012) · KI - Künstliche Intelligenz · doi:10.1007/s13218-012-0204-5 · Cited by 392
Introduction to spiking neural networks: Information processing, learning and applications
Ponulak, 2., & Kasinski, A. (2011) · Acta Neurobiologiae Experimentalis · doi:10.55782/ane-2011-1862 · Cited by 368
Randomness in neural networks: an overview
Scardapane, S., & Wang, D. (2017) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1200 · Cited by 352
Harnessing Disordered-Ensemble Quantum Dynamics for Machine Learning
Fujii, K., & Nakajima, K. (2017) · Physical Review Applied · doi:10.1103/physrevapplied.8.024030 · Cited by 337
Towards a theoretical foundation for morphological computation with compliant bodies
Hauser, H., Ijspeert, A. J., Füchslin, R. M., Pfeifer, R., & Maass, W. (2011) · Biological Cybernetics · doi:10.1007/s00422-012-0471-0 · Cited by 330
Information processing via physical soft body
Nakajima, K., Hauser, H., Li, T., & Pfeifer, R. (2015) · Scientific Reports · doi:10.1038/srep10487 · Cited by 324
Computing with Spiking Neuron Networks
Paugam-Moisy, H., & Bohte, S. (2012) · Handbook of Natural Computing · doi:10.1007/978-3-540-92910-9_10 · Cited by 272
Induction Machine Condition Monitoring Using Neural Network Modeling
Su, H., & Chong, K. T. (2007) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2006.888786 · Cited by 271
Support Vector Echo-State Machine for Chaotic Time-Series Prediction
Shi, Z., & Han, M. (2007) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2006.885113 · Cited by 258
A dynamic artificial neural network model for forecasting time series events
Ghiassi, M., Saidane, H., & Zimbra, D. K. (2005) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2004.10.008 · Cited by 257
A soft body as a reservoir: case studies in a dynamic model of octopus-inspired soft robotic arm
Nakajima, K., Hauser, H., Kang, R., Guglielmino, E., Caldwell, D. G., & Pfeifer, R. (2013) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00091 · Cited by 180
Boosting Computational Power through Spatial Multiplexing in Quantum Reservoir Computing
Nakajima, K., Fujii, K., Negoro, M., Mitarai, K., & Kitagawa, M. (2019) · Physical Review Applied · doi:10.1103/physrevapplied.11.034021 · Cited by 153
Pseudo-Random Generator based on a Photonic Neuromorphic Physical Unclonable Function
Dermanis, D., Rizomiliotis, P., Bogris, A., & Mesaritakis, C. (2024) · arXiv · doi:10.48550/arxiv.2402.14876 · Cited by 4
Metadata
- Authors
- Atiya, A. F., & Parlos, A. G.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 2000-05
- Volume
- 11
- Issue
- 3
- Pages
- 697-709
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2169762136
- Cited by
- 521
- References
- 40
- FWCI
- 11.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.