The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Gradient calculations for dynamic recurrent neural networks: a survey
Abstract
Surveys learning algorithms for recurrent neural networks with hidden units and puts the various techniques into a common framework. The authors discuss fixed point learning algorithms, namely recurrent backpropagation and deterministic Boltzmann machines, and nonfixed point algorithms, namely backpropagation through time, Elman's history cutoff, and Jordan's output feedback architecture. Forward propagation, an on-line technique that uses adjoint equations, and variations thereof, are also discussed. In many cases, the unified presentation leads to generalizations of various sorts. The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones continues with some "tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks. The author presents some simulations, and at the end, addresses issues of computational complexity and learning speed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
GELB, A. (1974) · CERN Document Server (European Organization for Nuclear Research) · Cited by 6,405
“Neural” computation of decisions in optimization problems
Hopfield, J. J., & Tank, D. W. (1985) · Biological Cybernetics · doi:10.1007/bf00339943 · Cited by 6,031
(1991) · Springer eBooks · doi:10.1007/978-3-7091-2840-4 · Cited by 5,095
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Anderson, B. D. O., Moore, J. B., & Eslami, M. (1982) · IEEE Transactions on Systems Man and Cybernetics · Cited by 3,195
Advances in Neural Information Processing Systems 37
(2024) · doi:10.52202/079017 · Cited by 2,791
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,789
Beyond regression : new fools for prediction and analysis in the behavioral sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,739
Alexander, S. T. (1986) · Springer eBooks · doi:10.1007/978-1-4612-4978-8 · Cited by 2,646
Phoneme recognition using time-delay neural networks
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.21701 · Cited by 2,643
Cohen, M. A., & Grossberg, S. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313075 · Cited by 2,541
Learning regular sets from queries and counterexamples
Angluin, D. (1987) · Information and Computation · doi:10.1016/0890-5401(87)90052-6 · Cited by 2,114
How brains make chaos in order to make sense of the world
Skarda, C. A., & Freeman, W. J. (1987) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00047336 · Cited by 2,097
Advances in Neural Information Processing Systems 5
Zemel, R. S., Williams, C. K. I., & Mozer, M. C. (1993) · Cited by 1,896
Increased rates of convergence through learning rate adaptation
Jacobs, R. A. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90003-2 · Cited by 1,806
On the identification of variances and adaptive Kalman filtering
Mehra, R. (1970) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1970.1099422 · Cited by 1,415
Marr, D., & Poggio, T. (1976) · Science · doi:10.1126/science.968482 · Cited by 1,377
Stationary and nonstationary learning characteristics of the LMS adaptive filter
Widrow, B., McCool, J. M., Larimore, M. G., & Johnson, C. R. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10286 · Cited by 1,268
Progress in supervised neural networks
Hush, D. R., & Horne, B. G. (1993) · IEEE Signal Processing Magazine · doi:10.1109/79.180705 · Cited by 1,247
Stationary and Nonstationary Learning Characteristics of the LMS Adaptive Filter
Widrow, B., McCool, J., Larimore, M. G., & Johnson, C. R. (1977) · Aspects of Signal Processing · doi:10.1007/978-94-010-1223-2_23 · Cited by 1,195
Analysis of hidden units in a layered network trained to classify sonar targets
Gorman, R. P., & Sejnowski, T. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90023-8 · Cited by 1,114
(2012) · SpringerReference · doi:10.1007/springerreference_117344 · Cited by 1,047
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 944
Generalization of back-propagation to recurrent neural networks
Pineda, F. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.2229 · Cited by 931 · cited 4×
“There are xedpoint learning algorithms (for details see [72], [73], [74], [75], or for a survey see [76]) 1Typically ( ) = (1+e ) 1, in which case 0( ) = ( )(1 ( )), or the scaled ( ) = tanh( ), in which case 0( ) = (1 + ( ))(1 ( )) = 1 2( ).”
MethodInfluentialAdvances in Neural Information Processing Systems 7
Tesauro, G., Touretzky, D. S., & Leen, T. K. (1995) · Cited by 869
Simulation of chaotic EEG patterns with a dynamic model of the olfactory system
Freeman, W. J. (1987) · Biological Cybernetics · doi:10.1007/bf00317988 · Cited by 828
Learning distributed representations of concepts.
Hinton, G. E. (1989) · eScholarship (California Digital Library) · Cited by 827
Neural Information Processing Systems
Anderson, D. Z. (1988) · neural information processing systems · Cited by 784
Learning State Space Trajectories in Recurrent Neural Networks
Pearlmutter, B. A. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.263 · Cited by 656
Self-citationGradient methods for the optimization of dynamical systems containing neural networks
Narendra, K. S., & Parthasarathy, K. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.80336 · Cited by 641
A time-delay neural network architecture for isolated word recognition
Lang, K. J., Waibel, A. H., & Hinton, G. E. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90044-l · Cited by 596
An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories
Williams, R. J., & Peng, J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.4.490 · Cited by 577
Rumelhart, D. E., & McClelland, J. L. (1986) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · Cited by 553
Hinton, G. E., & Sejnowski, J. (1983) · Cited by 498
A Steepest-Ascent Method for Solving Optimum Programming Problems
Bryson, A. E., & Denham, W. F. (1962) · Journal of Applied Mechanics · doi:10.1115/1.3640537 · Cited by 492
Finite State Automata and Simple Recurrent Networks
Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.372 · Cited by 458
Massively parallel parsing: A strongly interactive model of natural language interpretation
WALTZ, D., & POLLACK, J. (1985) · Cognitive Science · doi:10.1016/s0364-0213(85)80009-4 · Cited by 443
Backpropagation: Theory, architectures, and applications
Biron, P. V. (1997) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199701)48:1<88::aid-asi10>3.0.co;2-# · Cited by 435
Learning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398
Turing computability with neural nets
Siegelmann, H. T., & Sontag, E. D. (1991) · Applied Mathematics Letters · doi:10.1016/0893-9659(91)90080-f · Cited by 377
Analog computation via neural networks
Siegelmann, H. T., & Sontag, E. D. (1994) · Theoretical Computer Science · doi:10.1016/0304-3975(94)90178-3 · Cited by 355
Neural computation by concentrating information in time.
Tank, D. W., & Hopfield, J. J. (1987) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.84.7.1896 · Cited by 354
Equations of Motion from a Data Series.
Crutchfield, J. P., & McNamara, B. (1987) · Complex Systems · Cited by 294
Dynamic programming and the calculus of variations
Dreyfus, S. E. (1960) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(60)90024-x · Cited by 292
A learning rule for asynchronous perceptrons with feedback in a combinatorial environment
Almeida, L. B. (1990) · IEEE Press eBooks · Cited by 280
Applications of advances in nonlinear sensitivity analysis
Werbos, P. J. (2005) · Lecture Notes in Control and Information Sciences · doi:10.1007/bfb0006203 · Cited by 259
The gamma model—A new neural model for temporal processing
Vries, B. D., & Principe, J. C. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80035-8 · Cited by 254
Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization
Watrous, R. L. (1988) · ScholarlyCommons (University of Pennsylvania) · Cited by 242
FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling
Back, A. D., & Tsoi, A. C. (1991) · Neural Computation · doi:10.1162/neco.1991.3.3.375 · Cited by 210
Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space
Hinton, G. E. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.143 · Cited by 206
Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms
Nerrand, O., Roussel-Ragot, P., Personnaz, L., Dreyfus, G., & Marcos, S. (1993) · Neural Computation · doi:10.1162/neco.1993.5.2.165 · Cited by 183
Adapting bias by gradient descent: an incremental version of delta-bar-delta
Sutton, R. S. (1992) · Cited by 171
Generic constraints on underspecified target trajectories
Jordan, M. I. (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118584 · Cited by 167
Cooperating processes for low-level vision: A survey
Davis, L. S., & Rosenfeld, A. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90026-6 · Cited by 154
Dynamic recurrent neural networks
Pearlmutter, B. A. (2018) · Maynooth University ePrints and eTheses Archive (Maynooth University) · doi:10.1184/r1/6605018 · Cited by 132
Self-citationSchmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.243 · Cited by 130
Higher-order Boltzmann machines
B, T. J. S. A. (1986) · AIP conference proceedings · doi:10.1063/1.36246 · Cited by 121
Training recurrent networks using the extended Kalman filter
Williams, R. J. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.227335 · Cited by 118
Analysis of a cooperative stereo algorithm
Marr, D., Palm, G., & Poggio, T. (1978) · Biological Cybernetics · doi:10.1007/bf00344269 · Cited by 96
Learning symmetry groups with hidden units: Beyond the perceptron
B, T. J. S. A., Kienker, P. K., & Hinton, G. E. (1986) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(86)90245-9 · Cited by 94
Gain Adaptation Beats Least Squares
Sutton, R. S. (2006) · Cited by 92
Extracting and Learning an Unknown Grammar with Recurrent Neural Networks
Giles, C. L., MILLER, C. B., Chen, D., Sun, G.-Z., Chen, H. H., & Lee, Y. C. (1991) · neural information processing systems · Cited by 92
Jacobson, D. H. (1968) · Journal of Optimization Theory and Applications · doi:10.1007/bf00925746 · Cited by 91
Learning to Control an Unstable System with Forward Modeling
Jordan, M. I., & Jacobs, R. A. (1989) · Cited by 86
Continuous-time temporal back-propagation with adaptable time delays
Day, S. P., & Davenport, M. R. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.207622 · Cited by 83
Static and Dynamic Error Propagation Networks with Application to Speech Coding
Robinson, A. J., & Fallside, F. (1987) · Cited by 79
A learning algorithm for analog, fully recurrent neural networks
Gherrity (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118645 · Cited by 78
Renals, S. J., & Rohwer, R. (1990) · Journal of Statistical Physics · doi:10.1007/bf01026553 · Cited by 73
Shape recognition and illusory conjunctions
Hinton, G. E., & Lang, K. (1985) · Cited by 66
Relaxation and its Role in Vision
Hinton, G. E. (1977) · ERA · Cited by 64
Learning algorithms for oscillatory networks with gap junctions and membrane currents
Rowat, P. F., & Selverston, A. I. (1991) · Network Computation in Neural Systems · doi:10.1088/0954-898x_2_1_002 · Cited by 62
A Theory for Neural Networks with Time Delays
de Vries, B., & Príncipe, J. C. (1990) · neural information processing systems · Cited by 62
BPS: a learning algorithm for capturing the dynamic nature of speech
Gori, M., Bengio, & Mori, R. D. (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118276 · Cited by 60
Higher-order Boltzmann machines
Sejnowski, T. J. (1987) · Cited by 59
Contrastive Learning and Neural Oscillations
Baldi, P., & Pineda, F. J. (1991) · Neural Computation · doi:10.1162/neco.1991.3.4.526 · Cited by 58
Fool's Gold: Extracting Finite State Machines from Recurrent Network Dynamics
Kolen, J. F. (1993) · neural information processing systems · Cited by 57
The “moving targets” training algorithm
Rohwer, R. (1990) · Lecture notes in computer science · doi:10.1007/3-540-52255-7_31 · Cited by 57
A Dynamic Neural Network Model of Sensorimotor Transformations in the Leech
Lockery, S. R., Fang, Y., & B, T. J. S. A. (1990) · Neural Computation · doi:10.1162/neco.1990.2.3.274 · Cited by 46
Nonlinear dynamics of artificial neural systems
Maxwell, T., Giles, C. L., Lee, Y. C., & Chen, H. H. (1986) · AIP conference proceedings · doi:10.1063/1.36227 · Cited by 44
Atiya, A. F. (1987) · CaltechAUTHORS (California Institute of Technology) · Cited by 40
Faster Learning for Dynamic Recurrent Backpropagation
Fang, Y., & B, T. J. S. A. (1990) · Neural Computation · doi:10.1162/neco.1990.2.3.270 · Cited by 38
Asymptotic Convergence of Backpropagation
Tesauro, G. J., He, Y. B., & Ahmad, S. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.382 · Cited by 36
Learning Unambiguous Reduced Sequence Descriptions
Schmidhuber, J. (1991) · Cited by 33
A Connectionist Symbol Manipulator That Discovers the Structure of Context-Free Languages
Mozer, M. C., & Das, S. (1993) · Cited by 32
Dimensionality Reduction and Prior Knowledge in E-Set Recognition
Lang, K., & Hinton, G. E. (1989) · neural information processing systems · Cited by 31
Adaptive Soft Weight Tying using Gaussian Mixtures
Nowlan, S. J., & Hinton, G. E. (1991) · neural information processing systems · Cited by 31
Nonlinear prediction of speech signals using memory neuron networks
Poddar, P., & Unnikrishnan, K. P. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1991.239502 · Cited by 22
Deterministic Boltzmann Learning in Networks with Asymmetric Connectivity
Galland, C. C., & Hinton, G. E. (1991) · Elsevier eBooks · doi:10.1016/b978-1-4832-1448-1.50006-8 · Cited by 18
Learning of stable states in stochastic asymmetric networks
Allen, R. B., & Alspector, J. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80235 · Cited by 15
Using relaxation to find a puppet
Hinton, G. E. (1976) · Artificial Intelligence and the Simulation of Behaviour · Cited by 14
Cooperative algorithms for solving random-dot stereograms
Szeliski, R. (1986) · Cited by 9
Cited by
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025
“Methods for controllingthe dynamics of recurrent neural networks through adaptive mechanisms are reviewed in Pearlmutter (1995).”
Neural ordinary differential equations
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018) · arXiv · doi:10.48550/arxiv.1806.07366 · Cited by 2,097
A Critical Review of Recurrent Neural Networks for Sequence Learning
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 1,996
Metadata
- Authors
- Pearlmutter, B. A.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 1995
- Volume
- 6
- Issue
- 5
- Pages
- 1212-1228
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2154890045
- Cited by
- 582
- References
- 103
- FWCI
- 13.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.